Cluster version | Install command |
---|---|
v3.6 and v3.5 | pip install sparkmagic0.12.7 |
v3.4 | pip install sparkmagic0.2.3 |
ipywidgets
is properly installed by running the following command:sparkmagic
is installed by entering the following command:Kernel | Command |
---|---|
Spark | jupyter-kernelspec install sparkmagic/kernels/sparkkernel |
SparkR | jupyter-kernelspec install sparkmagic/kernels/sparkrkernel |
PySpark | jupyter-kernelspec install sparkmagic/kernels/pysparkkernel |
PySpark3 | jupyter-kernelspec install sparkmagic/kernels/pyspark3kernel |
.sparkmagic
, create a file called config.json and add the following JSON snippet inside it.Template value | New value |
---|---|
{USERNAME} | Cluster login, default is admin . |
{CLUSTERDNSNAME} | Cluster name |
{BASE64ENCODEDPASSWORD} | A base64 encoded password for your actual password. You can generate a base64 password at https://www.url-encode-decode.com/base64-encode-decode/. |
'livy_server_heartbeat_timeout_seconds': 60 | Keep if using sparkmagic 0.12.7 (clusters v3.5 and v3.6). If using sparkmagic 0.2.3 (clusters v3.4), replace with 'should_heartbeat': true . |
livy.server.interactive.heartbeat.timeout
to 0
from the Ambari UI. For clusters v3.5, if you do not set the 3.5 configuration above, the session will not be deleted.TypeError: __init__() got an unexpected keyword argument 'io_loop'
you may be experiencing a known issue with certain versions of Tornado. If so, stop the kernel and then downgrade your Tornado installation with the following command: pip install tornado4.5.3
.