When upgrading to version 1.4.0 you now have the option to enable large xattr. You can read more on XAttr in HopsFS XAttr Configuration. Large XAttrs are related to the ability to search Feature Groups and Trainig Datasets by their component Features. You can read more about this in Search by Feature Components
If you want to modify the XAttr size values, you can add this to a cluster definition:
hops:
xattrs:
max-xattrs-per-inode: 127
max-xattr-size: 3442755
If you want to modify these values on a live cluster, after you have finished the upgrade/installatiation, you can do this in the /srv/hops/hadoop/etc/hadoop/hdfs-site.xml
. You can modify the Modify the following properties:
<property>
<name>dfs.namenode.xattrs.enabled</name>
<value>...</value>
</property>
<property>
<name>dfs.namenode.fs-limits.max-xattrs-per-inode</name>
<value>...</value>
</property>
<property>
<name>dfs.namenode.fs-limits.max-xattr-size</name>
<value>...</value>
</property>
Hopsworks back end mechanism for managing projects’ Python environment is now more scalable and robust by moving to Docker based Anaconda environments. That means Python environments of existing projects will default back to the base Python environment provided by Hopsworks, so all additional Python libraries that were installed will be lost.
Users can find a YAML file that contains the exported Anaconda environment of the project including any user-installed libraries. This file can be found in the Resources dataset with a name python_env_<project>.yml where <project> is the project name. Users can use this YAML file to refer to previously installed Python libraries and then manually install them through the UI or the REST API. Importing the exported environment is discouraged as the base environment has been modified from the previous version and certain libraries such as TensorFlow will malfunction.
As of this version, Hopsworks projects are shipped by default with TensorFlow 2. TensorFlow 1.x is no longer supported automatically configured into the python dependencies of projects..