Hopsworks aggregates the logs of the applications launched and uses
Kibana for visualization and discovery. To get access to your logs,
upon a job completion, click on the Job UI
button and then on
Logs
tab. A sample output for the SparkPi job looks like the
following.
At the top section of the left-hand side of the main screen are the
selected fields of the log that are shown on the main page. At the
bottom section are the rest of the available indexed fields. On the
top is the search bar where you can search for specific fields. For
example if you want to preview the warning messages type
priority=WARN
. On the right side of the search bar you can save
your search query and load it later.
Also you can visualize certain fields of your logs by clicking on the Visualize button. For example, assume we want to make a pie chart of the severity of the log messages.
The pie chart should look like the following. On the top right-hand side is the legend. Hopefully most of your job’s messages will be INFO and a few WARN! On the top right corner is the visualization menu where you can save, load or share the current chart.