{
"/interpreter/livy.html": {
"title": "Livy Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Livy Interpreter for Apache ZeppelinOverviewLivy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN.Interactive Scala, Python and R shellsBatch submissions in Scala, Java, PythonMulti users can share the same server (impersonation support)Can be used for submitting jobs from anywhere with RESTDoes not require any code change to your programsRequirementsAdditional requirements for the Livy interpreter are:Spark 1.3 or above.Livy server.ConfigurationWe added some common configurations for spark, and you can set any configuration you want.You can find all Spark configurations in here.And instead of starting property with spark. it should be replaced with livy.spark..Example: spark.driver.memory to livy.spark.driver.memory Property Default Description zeppelin.livy.url http://localhost:8998 URL where livy server is running zeppelin.livy.spark.sql.maxResult 1000 Max number of Spark SQL result to display. zeppelin.livy.spark.sql.field.truncate true Whether to truncate field values longer than 20 characters or not zeppelin.livy.session.create_timeout 120 Timeout in seconds for session creation zeppelin.livy.displayAppInfo true Whether to display app info zeppelin.livy.pull_status.interval.millis 1000 The interval for checking paragraph execution status livy.spark.driver.cores Driver cores. ex) 1, 2. livy.spark.driver.memory Driver memory. ex) 512m, 32g. livy.spark.executor.instances Executor instances. ex) 1, 4. livy.spark.executor.cores Num cores per executor. ex) 1, 4. livy.spark.executor.memory Executor memory per worker instance. ex) 512m, 32g. livy.spark.dynamicAllocation.enabled Use dynamic resource allocation. ex) True, False. livy.spark.dynamicAllocation.cachedExecutorIdleTimeout Remove an executor which has cached data blocks. livy.spark.dynamicAllocation.minExecutors Lower bound for the number of executors. livy.spark.dynamicAllocation.initialExecutors Initial number of executors to run. livy.spark.dynamicAllocation.maxExecutors Upper bound for the number of executors. livy.spark.jars.packages Adding extra libraries to livy interpreter zeppelin.livy.ssl.trustStore client trustStore file. Used when livy ssl is enabled zeppelin.livy.ssl.trustStorePassword password for trustStore file. Used when livy ssl is enabled zeppelin.livy.http.headers key_1: value_1; key_2: value_2 custom http headers when calling livy rest api. Each http header is separated by `;`, and each header is one key value pair where key value is separated by `:` We remove livy.spark.master in zeppelin-0.7. Because we sugguest user to use livy 0.3 in zeppelin-0.7. And livy 0.3 don't allow to specify livy.spark.master, it enfornce yarn-cluster mode.Adding External librariesYou can load dynamic library to livy interpreter by set livy.spark.jars.packages property to comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. The format for the coordinates should be groupId:artifactId:version.Example Property Example Description livy.spark.jars.packages io.spray:spray-json_2.10:1.3.1 Adding extra libraries to livy interpreter How to useBasically, you can usespark%livy.sparksc.versionpyspark%livy.pysparkprint "1"sparkR%livy.sparkrhello <- function( name ) { sprintf( "Hello, %s", name );}hello("livy")ImpersonationWhen Zeppelin server is running with authentication enabled,then this interpreter utilizes Livy’s user impersonation featurei.e. sends extra parameter for creating and running a session ("proxyUser": "${loggedInUser}").This is particularly useful when multi users are sharing a Notebook server.Apply Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form. Form templates is only avalible for livy sql interpreter.%livy.sqlselect * from products where ${product_id=1}And creating dynamic formst programmatically is not feasible in livy interpreter, because ZeppelinContext is not available in livy interpreter.Shared SparkContextStarting from livy 0.5 which is supported by Zeppelin 0.8.0, SparkContext is shared between scala, python, r and sql.That means you can query the table via %livy.sql when this table is registered in %livy.spark, %livy.pyspark, $livy.sparkr.FAQLivy debugging: If you see any of these in error consoleConnect to livyhost:8998 [livyhost/127.0.0.1, livyhost/0:0:0:0:0:0:0:1] failed: Connection refusedLooks like the livy server is not up yet or the config is wrongException: Session not found, Livy server would have restarted, or lost session.The session would have timed out, you may need to restart the interpreter.Blacklisted configuration values in session config: spark.masterEdit conf/spark-blacklist.conf file in livy server and comment out #spark.master line.If you choose to work on livy in apps/spark/java directory in https://github.com/cloudera/hue,copy spark-user-configurable-options.template to spark-user-configurable-options.conf file in livy server and comment out #spark.master.",
"url": " /interpreter/livy.html",
"group": "interpreter",
"excerpt": "Livy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN."
}
,
"/interpreter/pig.html": {
"title": "Pig Interpreter for Apache Zeppelin",
"content" : "Pig Interpreter for Apache ZeppelinOverviewApache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets.Supported interpreter type%pig.script (default Pig interpreter, so you can use %pig)%pig.script is like the Pig grunt shell. Anything you can run in Pig grunt shell can be run in %pig.script interpreter, it is used for running Pig script where you don’t need to visualize the data, it is suitable for data munging. %pig.query%pig.query is a little different compared with %pig.script. It is used for exploratory data analysis via Pig latin where you can leverage Zeppelin’s visualization ability. There're 2 minor differences in the last statement between %pig.script and %pig.queryNo pig alias in the last statement in %pig.query (read the examples below).The last statement must be in single line in %pig.queryHow to useHow to setup Pig execution modes.Local ModeSet zeppelin.pig.execType as local.MapReduce ModeSet zeppelin.pig.execType as mapreduce. HADOOP_CONF_DIR needs to be specified in ZEPPELIN_HOME/conf/zeppelin-env.sh.Tez Local ModeOnly Tez 0.7 is supported. Set zeppelin.pig.execType as tez_local.Tez ModeOnly Tez 0.7 is supported. Set zeppelin.pig.execType as tez. HADOOP_CONF_DIR and TEZ_CONF_DIR needs to be specified in ZEPPELIN_HOME/conf/zeppelin-env.sh.Spark Local ModeOnly Spark 1.6.x is supported, by default it is Spark 1.6.3. Set zeppelin.pig.execType as spark_local.Spark ModeOnly Spark 1.6.x is supported, by default it is Spark 1.6.3. Set zeppelin.pig.execType as spark. For now, only yarn-client mode is supported. To enable it, you need to set property SPARK_MASTER to yarn-client and set SPARK_JAR to the spark assembly jar.How to choose custom Spark VersionBy default, Pig Interpreter would use Spark 1.6.3 built with scala 2.10, if you want to use another spark version or scala version, you need to rebuild Zeppelin by specifying the custom Spark version via -Dpig.spark.version= and scala version via -Dpig.scala.version= in the maven build command.How to configure interpreterAt the Interpreters menu, you have to create a new Pig interpreter. Pig interpreter has below properties by default.And you can set any Pig properties here which will be passed to Pig engine. (like tez.queue.name & mapred.job.queue.name).Besides, we use paragraph title as job name if it exists, else use the last line of Pig script. So you can use that to find app running in YARN RM UI. Property Default Description zeppelin.pig.execType mapreduce Execution mode for pig runtime. local | mapreduce | tez_local | tez | spark_local | spark zeppelin.pig.includeJobStats false whether display jobStats info in %pig.script zeppelin.pig.maxResult 1000 max row number displayed in %pig.query tez.queue.name default queue name for tez engine mapred.job.queue.name default queue name for mapreduce engine SPARK_MASTER local local | yarn-client SPARK_JAR The spark assembly jar, both jar in local or hdfs is supported. Put it on hdfs could have performance benefit Examplepig%pigbankText = load 'bank.csv' using PigStorage(';');bank = foreach bankText generate $0 as age, $1 as job, $2 as marital, $3 as education, $5 as balance; bank = filter bank by age != '"age"';bank = foreach bank generate (int)age, REPLACE(job,'"','') as job, REPLACE(marital, '"', '') as marital, (int)(REPLACE(balance, '"', '')) as balance;store bank into 'clean_bank.csv' using PigStorage(';'); -- this statement is optional, it just show you that most of time %pig.script is used for data munging before querying the data. pig.queryGet the number of each age where age is less than 30%pig.querybank_data = filter bank by age < 30;b = group bank_data by age;foreach b generate group, COUNT($1);The same as above, but use dynamic text form so that use can specify the variable maxAge in textbox. (See screenshot below). Dynamic form is a very cool feature of Zeppelin, you can refer this link) for details.%pig.querybank_data = filter bank by age < ${maxAge=40};b = group bank_data by age;foreach b generate group, COUNT($1) as count;Get the number of each age for specific marital type, also use dynamic form here. User can choose the marital type in the dropdown list (see screenshot below).%pig.querybank_data = filter bank by marital=='${marital=single,single|divorced|married}';b = group bank_data by age;foreach b generate group, COUNT($1) as count;The above examples are in the Pig tutorial note in Zeppelin, you can check that for details. Here's the screenshot.Data is shared between %pig and %pig.query, so that you can do some common work in %pig, and do different kinds of query based on the data of %pig. Besides, we recommend you to specify alias explicitly so that the visualization can display the column name correctly. In the above example 2 and 3 of %pig.query, we name COUNT($1) as count. If you don't do this, then we will name it using position. E.g. in the above first example of %pig.query, we will use col_1 in chart to represent COUNT($1).",
"url": " /interpreter/pig.html",
"group": "manual",
"excerpt": "Apache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs."
}
,
"/interpreter/markdown.html": {
"title": "Markdown Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Markdown Interpreter for Apache ZeppelinOverviewMarkdown is a plain text formatting syntax designed so that it can be converted to HTML.Apache Zeppelin uses pegdown and markdown4j as markdown parsers.In Zeppelin notebook, you can use %md in the beginning of a paragraph to invoke the Markdown interpreter and generate static html from Markdown plain text.In Zeppelin, Markdown interpreter is enabled by default and uses the pegdown parser.ExampleThe following example demonstrates the basic usage of Markdown in a Zeppelin notebook.Mathematical expressionMarkdown interpreter leverages %html display system internally. That means you can mix mathematical expressions with markdown syntax. For more information, please see Mathematical Expression section.Configuration Name Default Value Description markdown.parser.type pegdown Markdown Parser Type. Available values: pegdown, markdown4j. Pegdown Parserpegdown parser provides github flavored markdown.pegdown parser provides YUML and Websequence plugins also. Markdown4j ParserSince pegdown parser is more accurate and provides much more markdown syntax markdown4j option might be removed later. But keep this parser for the backward compatibility.",
"url": " /interpreter/markdown.html",
"group": "interpreter",
"excerpt": "Markdown is a plain text formatting syntax designed so that it can be converted to HTML. Apache Zeppelin uses markdown4j."
}
,
"/interpreter/mahout.html": {
"title": "Mahout Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Mahout Interpreter for Apache ZeppelinInstallationApache Mahout is a collection of packages that enable machine learning and matrix algebra on underlying engines such as Apache Flink or Apache Spark. A convenience script for creating and configuring two Mahout enabled interpreters exists. The %sparkMahout and %flinkMahout interpreters do not exist by default but can be easily created using this script. Easy InstallationTo quickly and easily get up and running using Apache Mahout, run the following command from the top-level directory of the Zeppelin install:python scripts/mahout/add_mahout.pyThis will create the %sparkMahout and %flinkMahout interpreters, and restart Zeppelin.Advanced InstallationThe add_mahout.py script contains several command line arguments for advanced users. Argument Description Example --zeppelin_home This is the path to the Zeppelin installation. This flag is not needed if the script is run from the top-level installation directory or from the zeppelin/scripts/mahout directory. /path/to/zeppelin --mahout_home If the user has already installed Mahout, this flag can set the path to MAHOUT_HOME. If this is set, downloading Mahout will be skipped. /path/to/mahout_home --restart_later Restarting is necessary for updates to take effect. By default the script will restart Zeppelin for you. Restart will be skipped if this flag is set. NA --force_download This flag will force the script to re-download the binary even if it already exists. This is useful for previously failed downloads. NA --overwrite_existing This flag will force the script to overwrite existing %sparkMahout and %flinkMahout interpreters. Useful when you want to just start over. NA NOTE 1: Apache Mahout at this time only supports Spark 1.5 and Spark 1.6 and Scala 2.10. If the user is using another version of Spark (e.g. 2.0), the %sparkMahout will likely not work. The %flinkMahout interpreter will still work and the user is encouraged to develop with that engine as the code can be ported via copy and paste, as is evidenced by the tutorial notebook.NOTE 2: If using Apache Flink in cluster mode, the following libraries will also need to be coppied to ${FLINK_HOME}/lib- mahout-math-0.12.2.jar- mahout-math-scala2.10-0.12.2.jar- mahout-flink2.10-0.12.2.jar- mahout-hdfs-0.12.2.jar- com.google.guava:guava:14.0.1OverviewThe Apache Mahout™ project's goal is to build an environment for quickly creating scalable performant machine learning applications.Apache Mahout software provides three major features:A simple and extensible programming environment and framework for building scalable algorithmsA wide variety of premade algorithms for Scala + Apache Spark, H2O, Apache FlinkSamsara, a vector math experimentation environment with R-like syntax which works at scaleIn other words:Apache Mahout provides a unified API for quickly creating machine learning algorithms on a variety of engines.How to useWhen starting a session with Apache Mahout, depending on which engine you are using (Spark or Flink), a few imports must be made and a Distributed Context must be declared. Copy and paste the following code and run once to get started.Flink%flinkMahoutimport org.apache.flink.api.scala._import org.apache.mahout.math.drm._import org.apache.mahout.math.drm.RLikeDrmOps._import org.apache.mahout.flinkbindings._import org.apache.mahout.math._import scalabindings._import RLikeOps._implicit val ctx = new FlinkDistributedContext(benv)Spark%sparkMahoutimport org.apache.mahout.math._import org.apache.mahout.math.scalabindings._import org.apache.mahout.math.drm._import org.apache.mahout.math.scalabindings.RLikeOps._import org.apache.mahout.math.drm.RLikeDrmOps._import org.apache.mahout.sparkbindings._implicit val sdc: org.apache.mahout.sparkbindings.SparkDistributedContext = sc2sdc(sc)Same Code, Different EnginesAfter importing and setting up the distributed context, the Mahout R-Like DSL is consistent across engines. The following code will run in both %flinkMahout and %sparkMahoutval drmData = drmParallelize(dense( (2, 2, 10.5, 10, 29.509541), // Apple Cinnamon Cheerios (1, 2, 12, 12, 18.042851), // Cap'n'Crunch (1, 1, 12, 13, 22.736446), // Cocoa Puffs (2, 1, 11, 13, 32.207582), // Froot Loops (1, 2, 12, 11, 21.871292), // Honey Graham Ohs (2, 1, 16, 8, 36.187559), // Wheaties Honey Gold (6, 2, 17, 1, 50.764999), // Cheerios (3, 2, 13, 7, 40.400208), // Clusters (3, 3, 13, 4, 45.811716)), numPartitions = 2)drmData.collect(::, 0 until 4)val drmX = drmData(::, 0 until 4)val y = drmData.collect(::, 4)val drmXtX = drmX.t %*% drmXval drmXty = drmX.t %*% yval XtX = drmXtX.collectval Xty = drmXty.collect(::, 0)val beta = solve(XtX, Xty)Leveraging Resource Pools and R for VisualizationResource Pools are a powerful Zeppelin feature that lets us share information between interpreters. A fun trick is to take the output of our work in Mahout and analyze it in other languages.Setting up a Resource Pool in FlinkIn Spark based interpreters resource pools are accessed via the ZeppelinContext API. To put and get things from the resource pool one can be done simpleval myVal = 1z.put("foo", myVal)val myFetchedVal = z.get("foo")To add this functionality to a Flink based interpreter we declare the follwoing%flinkMahoutimport org.apache.zeppelin.interpreter.InterpreterContextval z = InterpreterContext.get().getResourcePool()Now we can access the resource pool in a consistent manner from the %flinkMahout interpreter.Passing a variable from Mahout to R and PlottingIn this simple example, we use Mahout (on Flink or Spark, the code is the same) to create a random matrix and then take the Sin of each element. We then randomly sample the matrix and create a tab separated string. Finally we pass that string to R where it is read as a .tsv file, and a DataFrame is created and plotted using native R plotting libraries.val mxRnd = Matrices.symmetricUniformView(5000, 2, 1234)val drmRand = drmParallelize(mxRnd)val drmSin = drmRand.mapBlock() {case (keys, block) => val blockB = block.like() for (i <- 0 until block.nrow) { blockB(i, 0) = block(i, 0) blockB(i, 1) = Math.sin((block(i, 0) * 8)) } keys -> blockB}z.put("sinDrm", org.apache.mahout.math.drm.drmSampleToTSV(drmSin, 0.85))And then in an R paragraph...%spark.r {"imageWidth": "400px"}library("ggplot2")sinStr = z.get("flinkSinDrm")data <- read.table(text= sinStr, sep="t", header=FALSE)plot(data, col="red")",
"url": " /interpreter/mahout.html",
"group": "interpreter",
"excerpt": "Apache Mahout provides a unified API (the R-Like Scala DSL) for quickly creating machine learning algorithms on a variety of engines."
}
,
"/interpreter/spark.html": {
"title": "Apache Spark Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Spark Interpreter for Apache ZeppelinOverviewApache Spark is a fast and general-purpose cluster computing system.It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs.Apache Spark is supported in Zeppelin with Spark interpreter group which consists of below five interpreters. Name Class Description %spark SparkInterpreter Creates a SparkContext and provides a Scala environment %spark.pyspark PySparkInterpreter Provides a Python environment %spark.r SparkRInterpreter Provides an R environment with SparkR support %spark.sql SparkSQLInterpreter Provides a SQL environment %spark.dep DepInterpreter Dependency loader ConfigurationThe Spark interpreter can be configured with properties provided by Zeppelin.You can also set other Spark properties which are not listed in the table. For a list of additional properties, refer to Spark Available Properties. Property Default Description args Spark commandline args master local[*] Spark master uri. ex) spark://masterhost:7077 spark.app.name Zeppelin The name of spark application. spark.cores.max Total number of cores to use. Empty value uses all available core. spark.executor.memory 1g Executor memory per worker instance. ex) 512m, 32g zeppelin.dep.additionalRemoteRepository spark-packages, http://dl.bintray.com/spark-packages/maven, false; A list of id,remote-repository-URL,is-snapshot; for each remote repository. zeppelin.dep.localrepo local-repo Local repository for dependency loader PYSPARK_PYTHON python Python binary executable to use for PySpark in both driver and workers (default is python). Property spark.pyspark.python take precedence if it is set PYSPARK_DRIVER_PYTHON python Python binary executable to use for PySpark in driver only (default is PYSPARK_PYTHON). Property spark.pyspark.driver.python take precedence if it is set zeppelin.spark.concurrentSQL false Execute multiple SQL concurrently if set true. zeppelin.spark.maxResult 1000 Max number of Spark SQL result to display. zeppelin.spark.printREPLOutput true Print REPL output zeppelin.spark.useHiveContext true Use HiveContext instead of SQLContext if it is true. zeppelin.spark.importImplicit true Import implicits, UDF collection, and sql if set true. zeppelin.spark.enableSupportedVersionCheck true Do not change - developer only setting, not for production use zeppelin.spark.sql.interpolation false Enable ZeppelinContext variable interpolation into paragraph text zeppelin.spark.uiWebUrl Overrides Spark UI default URL. Value should be a full URL (ex: http://{hostName}/{uniquePath} Without any configuration, Spark interpreter works out of box in local mode. But if you want to connect to your Spark cluster, you'll need to follow below two simple steps.1. Export SPARK_HOMEIn conf/zeppelin-env.sh, export SPARK_HOME environment variable with your Spark installation path.For example,export SPARK_HOME=/usr/lib/sparkYou can optionally set more environment variables# set hadoop conf direxport HADOOP_CONF_DIR=/usr/lib/hadoop# set options to pass spark-submit commandexport SPARK_SUBMIT_OPTIONS="--packages com.databricks:spark-csv_2.10:1.2.0"# extra classpath. e.g. set classpath for hive-site.xmlexport ZEPPELIN_INTP_CLASSPATH_OVERRIDES=/etc/hive/confFor Windows, ensure you have winutils.exe in %HADOOP_HOME%bin. Please see Problems running Hadoop on Windows for the details.2. Set master in Interpreter menuAfter start Zeppelin, go to Interpreter menu and edit master property in your Spark interpreter setting. The value may vary depending on your Spark cluster deployment type.For example,local[*] in local modespark://master:7077 in standalone clusteryarn-client in Yarn client modeyarn-cluster in Yarn cluster modemesos://host:5050 in Mesos clusterThat's it. Zeppelin will work with any version of Spark and any deployment type without rebuilding Zeppelin in this way.For the further information about Spark & Zeppelin version compatibility, please refer to "Available Interpreters" section in Zeppelin download page.Note that without exporting SPARK_HOME, it's running in local mode with included version of Spark. The included version may vary depending on the build profile.3. Yarn modeZeppelin support both yarn client and yarn cluster mode (yarn cluster mode is supported from 0.8.0). For yarn mode, you must specify SPARK_HOME & HADOOP_CONF_DIR.You can either specify them in zeppelin-env.sh, or in interpreter setting page. Specifying them in zeppelin-env.sh means you can use only one version of spark & hadoop. Specifying themin interpreter setting page means you can use multiple versions of spark & hadoop in one zeppelin instance.4. New Version of SparkInterpreterThere's one new version of SparkInterpreter with better spark support and code completion starting from Zeppelin 0.8.0. We enable it by default, but user can still use the old version of SparkInterpreter by setting zeppelin.spark.useNew as false in its interpreter setting.SparkContext, SQLContext, SparkSession, ZeppelinContextSparkContext, SQLContext and ZeppelinContext are automatically created and exposed as variable names sc, sqlContext and z, respectively, in Scala, Python and R environments.Staring from 0.6.1 SparkSession is available as variable spark when you are using Spark 2.x.Note that Scala/Python/R environment shares the same SparkContext, SQLContext and ZeppelinContext instance. How to pass property to SparkConfThere're 2 kinds of properties that would be passed to SparkConfStandard spark property (prefix with spark.). e.g. spark.executor.memory will be passed to SparkConfNon-standard spark property (prefix with zeppelin.spark.). e.g. zeppelin.spark.property_1, property_1 will be passed to SparkConfDependency ManagementThere are two ways to load external libraries in Spark interpreter. First is using interpreter setting menu and second is loading Spark properties.1. Setting Dependencies via Interpreter SettingPlease see Dependency Management for the details.2. Loading Spark PropertiesOnce SPARK_HOME is set in conf/zeppelin-env.sh, Zeppelin uses spark-submit as spark interpreter runner. spark-submit supports two ways to load configurations.The first is command line options such as --master and Zeppelin can pass these options to spark-submit by exporting SPARK_SUBMIT_OPTIONS in conf/zeppelin-env.sh. Second is reading configuration options from SPARK_HOME/conf/spark-defaults.conf. Spark properties that user can set to distribute libraries are: spark-defaults.conf SPARK_SUBMIT_OPTIONS Description spark.jars --jars Comma-separated list of local jars to include on the driver and executor classpaths. spark.jars.packages --packages Comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. Will search the local maven repo, then maven central and any additional remote repositories given by --repositories. The format for the coordinates should be groupId:artifactId:version. spark.files --files Comma-separated list of files to be placed in the working directory of each executor. Here are few examples:SPARK_SUBMIT_OPTIONS in conf/zeppelin-env.shexport SPARK_SUBMIT_OPTIONS="--packages com.databricks:spark-csv_2.10:1.2.0 --jars /path/mylib1.jar,/path/mylib2.jar --files /path/mylib1.py,/path/mylib2.zip,/path/mylib3.egg"SPARK_HOME/conf/spark-defaults.confspark.jars /path/mylib1.jar,/path/mylib2.jarspark.jars.packages com.databricks:spark-csv_2.10:1.2.0spark.files /path/mylib1.py,/path/mylib2.egg,/path/mylib3.zip3. Dynamic Dependency Loading via %spark.dep interpreterNote: %spark.dep interpreter loads libraries to %spark and %spark.pyspark but not to %spark.sql interpreter. So we recommend you to use the first option instead.When your code requires external library, instead of doing download/copy/restart Zeppelin, you can easily do following jobs using %spark.dep interpreter.Load libraries recursively from maven repositoryLoad libraries from local filesystemAdd additional maven repositoryAutomatically add libraries to SparkCluster (You can turn off)Dep interpreter leverages Scala environment. So you can write any Scala code here.Note that %spark.dep interpreter should be used before %spark, %spark.pyspark, %spark.sql.Here's usages.%spark.depz.reset() // clean up previously added artifact and repository// add maven repositoryz.addRepo("RepoName").url("RepoURL")// add maven snapshot repositoryz.addRepo("RepoName").url("RepoURL").snapshot()// add credentials for private maven repositoryz.addRepo("RepoName").url("RepoURL").username("username").password("password")// add artifact from filesystemz.load("/path/to.jar")// add artifact from maven repository, with no dependencyz.load("groupId:artifactId:version").excludeAll()// add artifact recursivelyz.load("groupId:artifactId:version")// add artifact recursively except comma separated GroupID:ArtifactId listz.load("groupId:artifactId:version").exclude("groupId:artifactId,groupId:artifactId, ...")// exclude with patternz.load("groupId:artifactId:version").exclude(*)z.load("groupId:artifactId:version").exclude("groupId:artifactId:*")z.load("groupId:artifactId:version").exclude("groupId:*")// local() skips adding artifact to spark clusters (skipping sc.addJar())z.load("groupId:artifactId:version").local()ZeppelinContextZeppelin automatically injects ZeppelinContext as variable z in your Scala/Python environment. ZeppelinContext provides some additional functions and utilities.See Zeppelin-Context for more details.Matplotlib Integration (pyspark)Both the python and pyspark interpreters have built-in support for inline visualization using matplotlib,a popular plotting library for python. More details can be found in the python interpreter documentation,since matplotlib support is identical. More advanced interactive plotting can be done with pyspark throughutilizing Zeppelin's built-in Angular Display System, as shown below:Interpreter setting optionYou can choose one of shared, scoped and isolated options wheh you configure Spark interpreter.Spark interpreter creates separated Scala compiler per each notebook but share a single SparkContext in scoped mode (experimental).It creates separated SparkContext per each notebook in isolated mode.IPython supportBy default, zeppelin would use IPython in pyspark when IPython is available, Otherwise it would fall back to the original PySpark implementation.If you don't want to use IPython, then you can set zeppelin.pyspark.useIPython as false in interpreter setting. For the IPython features, you can refer docPython InterpreterSetting up Zeppelin with KerberosLogical setup with Zeppelin, Kerberos Key Distribution Center (KDC), and Spark on YARN:Configuration SetupOn the server that Zeppelin is installed, install Kerberos client modules and configuration, krb5.conf.This is to make the server communicate with KDC.Set SPARK_HOME in [ZEPPELIN_HOME]/conf/zeppelin-env.sh to use spark-submit(Additionally, you might have to set export HADOOP_CONF_DIR=/etc/hadoop/conf)Add the two properties below to Spark configuration ([SPARK_HOME]/conf/spark-defaults.conf):spark.yarn.principalspark.yarn.keytabNOTE: If you do not have permission to access for the above spark-defaults.conf file, optionally, you can add the above lines to the Spark Interpreter setting through the Interpreter tab in the Zeppelin UI.That's it. Play with Zeppelin!",
"url": " /interpreter/spark.html",
"group": "interpreter",
"excerpt": "Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution engine."
}
,
"/interpreter/python.html": {
"title": "Python 2 & 3 Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Python 2 & 3 Interpreter for Apache ZeppelinConfiguration Property Default Description zeppelin.python python Path of the already installed Python binary (could be python2 or python3). If python is not in your $PATH you can set the absolute directory (example : /usr/bin/python) zeppelin.python.maxResult 1000 Max number of dataframe rows to display. Enabling Python InterpreterIn a notebook, to enable the Python interpreter, click on the Gear icon and select PythonUsing the Python InterpreterIn a paragraph, use %python to select the Python interpreter and then input all commands.The interpreter can only work if you already have python installed (the interpreter doesn't bring it own python binaries).To access the help, type help()Python environmentsDefaultBy default, PythonInterpreter will use python command defined in zeppelin.python property to run python process.The interpreter can use all modules already installed (with pip, easy_install...)CondaConda is an package management system and environment management system for python.%python.conda interpreter lets you change between environments.Usageget the Conda Infomation: %python.conda infolist the Conda environments: %python.conda env listcreate a conda enviornment: %python.conda create --name [ENV NAME]activate an environment (python interpreter will be restarted): %python.conda activate [ENV NAME]deactivate%python.conda deactivateget installed package list inside the current environment%python.conda listinstall package%python.conda install [PACKAGE NAME]uninstall package%python.conda uninstall [PACKAGE NAME]Docker%python.docker interpreter allows PythonInterpreter creates python process in a specified docker container.Usageactivate an environment%python.docker activate [Repository]%python.docker activate [Repository:Tag]%python.docker activate [Image Id]deactivate%python.docker deactivateHere is an example# activate latest tensorflow image as a python environment%python.docker activate gcr.io/tensorflow/tensorflow:latestUsing Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form inside your Python code.Zeppelin Dynamic Form can only be used if py4j Python library is installed in your system. If not, you can install it with pip install py4j.Example : %python### Input formprint (z.input("f1","defaultValue"))### Select formprint (z.select("f1",[("o1","1"),("o2","2")],"2"))### Checkbox formprint("".join(z.checkbox("f3", [("o1","1"), ("o2","2")],["1"])))Matplotlib integrationThe python interpreter can display matplotlib figures inline automatically using the pyplot module:%pythonimport matplotlib.pyplot as pltplt.plot([1, 2, 3])This is the recommended method for using matplotlib from within a Zeppelin notebook. The output of this command will by default be converted to HTML by implicitly making use of the %html magic. Additional configuration can be achieved using the builtin z.configure_mpl() method. For example, z.configure_mpl(width=400, height=300, fmt='svg')plt.plot([1, 2, 3])Will produce a 400x300 image in SVG format, which by default are normally 600x400 and PNG respectively. In the future, another option called angular can be used to make it possible to update a plot produced from one paragraph directly from another (the output will be %angular instead of %html). However, this feature is already available in the pyspark interpreter. More details can be found in the included "Zeppelin Tutorial: Python - matplotlib basic" tutorial notebook. If Zeppelin cannot find the matplotlib backend files (which should usually be found in $ZEPPELIN_HOME/interpreter/lib/python) in your PYTHONPATH, then the backend will automatically be set to agg, and the (otherwise deprecated) instructions below can be used for more limited inline plotting.If you are unable to load the inline backend, use z.show(plt):%pythonimport matplotlib.pyplot as pltplt.figure()(.. ..)z.show(plt)plt.close()The z.show() function can take optional parameters to adapt graph dimensions (width and height) as well as output format (png or optionally svg).%pythonz.show(plt, width='50px')z.show(plt, height='150px', fmt='svg')Pandas integrationApache Zeppelin Table Display System provides built-in data visualization capabilities. Python interpreter leverages it to visualize Pandas DataFrames though similar z.show() API, same as with Matplotlib integration.Example:import pandas as pdrates = pd.read_csv("bank.csv", sep=";")z.show(rates)SQL over Pandas DataFramesThere is a convenience %python.sql interpreter that matches Apache Spark experience in Zeppelin and enables usage of SQL language to query Pandas DataFrames and visualization of results though built-in Table Display System.Pre-requestsPandas pip install pandasPandaSQL pip install -U pandasqlIn case default binded interpreter is Python (first in the interpreter list, under the Gear Icon), you can just use it as %sql i.efirst paragraphimport pandas as pdrates = pd.read_csv("bank.csv", sep=";")next paragraph%sqlSELECT * FROM rates WHERE age < 40Otherwise it can be referred to as %python.sqlIPython SupportIPython is more powerful than the default python interpreter with extra functionality. You can use IPython with Python2 or Python3 which depends on which python you set zeppelin.python.Pre-requests- Jupyter `pip install jupyter`- grpcio `pip install grpcio`- protobuf `pip install protobuf`If you already install anaconda, then you just need to install grpcio as Jupyter is already included in anaconda. For grpcio version >= 1.12.0 you'll also need to install protobuf separately.In addition to all basic functions of the python interpreter, you can use all the IPython advanced features as you use it in Jupyter Notebook.e.g. Use IPython magic%python.ipython#python helprange?#timeit%timeit range(100)Use matplotlib %python.ipython%matplotlib inlineimport matplotlib.pyplot as pltprint("hello world")data=[1,2,3,4]plt.figure()plt.plot(data)We also make ZeppelinContext available in IPython Interpreter. You can use ZeppelinContext to create dynamic forms and display pandas DataFrame.e.g.Create dynamic formz.input(name='my_name', defaultValue='hello')Show pandas dataframeimport pandas as pddf = pd.DataFrame({'id':[1,2,3], 'name':['a','b','c']})z.show(df)By default, we would use IPython in %python.python if IPython is available. Otherwise it would fall back to the original Python implementation.If you don't want to use IPython, then you can set zeppelin.python.useIPython as false in interpreter setting.Technical descriptionFor in-depth technical details on current implementation please refer to python/README.md.Some features not yet implemented in the Python InterpreterInterrupt a paragraph execution (cancel() method) is currently only supported in Linux and MacOs. If interpreter runs in another operating system (for instance MS Windows) , interrupt a paragraph will close the whole interpreter. A JIRA ticket (ZEPPELIN-893) is opened to implement this feature in a next release of the interpreter.Progression bar in webUI (getProgress() method) is currently not implemented.Code-completion is currently not implemented.",
"url": " /interpreter/python.html",
"group": "interpreter",
"excerpt": "Python is a programming language that lets you work quickly and integrate systems more effectively."
}
,
"/interpreter/hive.html": {
"title": "Hive Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Hive Interpreter for Apache ZeppelinImportant NoticeHive Interpreter will be deprecated and merged into JDBC Interpreter. You can use Hive Interpreter by using JDBC Interpreter with same functionality. See the example below of settings and dependencies.Properties Property Value hive.driver org.apache.hive.jdbc.HiveDriver hive.url jdbc:hive2://localhost:10000 hive.user hiveUser hive.password hivePassword Dependencies Artifact Exclude org.apache.hive:hive-jdbc:0.14.0 org.apache.hadoop:hadoop-common:2.6.0 Configuration Property Default Description default.driver org.apache.hive.jdbc.HiveDriver Class path of JDBC driver default.url jdbc:hive2://localhost:10000 Url for connection default.user ( Optional ) Username of the connection default.password ( Optional ) Password of the connection default.xxx ( Optional ) Other properties used by the driver ${prefix}.driver Driver class path of %hive(${prefix}) ${prefix}.url Url of %hive(${prefix}) ${prefix}.user ( Optional ) Username of the connection of %hive(${prefix}) ${prefix}.password ( Optional ) Password of the connection of %hive(${prefix}) ${prefix}.xxx ( Optional ) Other properties used by the driver of %hive(${prefix}) This interpreter provides multiple configuration with ${prefix}. User can set a multiple connection properties by this prefix. It can be used like %hive(${prefix}).OverviewThe Apache Hive ™ data warehouse software facilitates querying and managing large datasets residing in distributed storage. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this language also allows traditional map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefficient to express this logic in HiveQL.How to useBasically, you can use%hiveselect * from my_table;or%hive(etl)-- 'etl' is a ${prefix}select * from my_table;You can also run multiple queries up to 10 by default. Changing these settings is not implemented yet.Apply Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form inside your queries. You can use both the text input and select form parameterization features.%hiveSELECT ${group_by}, count(*) as countFROM retail_demo.order_lineitems_pxfGROUP BY ${group_by=product_id,product_id|product_name|customer_id|store_id}ORDER BY count ${order=DESC,DESC|ASC}LIMIT ${limit=10};",
"url": " /interpreter/hive.html",
"group": "interpreter",
"excerpt": "Apache Hive data warehouse software facilitates querying and managing large datasets residing in distributed storage. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this..."
}
,
"/interpreter/ignite.html": {
"title": "Ignite Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Ignite Interpreter for Apache ZeppelinOverviewApache Ignite In-Memory Data Fabric is a high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash technologies.You can use Zeppelin to retrieve distributed data from cache using Ignite SQL interpreter. Moreover, Ignite interpreter allows you to execute any Scala code in cases when SQL doesn't fit to your requirements. For example, you can populate data into your caches or execute distributed computations.Installing and Running Ignite exampleIn order to use Ignite interpreters, you may install Apache Ignite in some simple steps:Ignite provides examples only with source or binary release. Download Ignite source release or binary release whatever you want. But you must download Ignite as the same version of Zeppelin's. If it is not, you can't use scala code on Zeppelin. The supported Ignite version is specified in Supported Interpreter table for each Zeppelin release. If you're using Zeppelin master branch, please see ignite.version in path/to/your-Zeppelin/ignite/pom.xml.Examples are shipped as a separate Maven project, so to start running you simply need to import provided <dest_dir>/apache-ignite-fabric-{version}-bin/examples/pom.xml file into your favourite IDE, such as Eclipse.In case of Eclipse, Eclipse -> File -> Import -> Existing Maven ProjectsSet examples directory path to Eclipse and select the pom.xml.Then start org.apache.ignite.examples.ExampleNodeStartup (or whatever you want) to run at least one or more ignite node. When you run example code, you may notice that the number of node is increase one by one.Tip. If you want to run Ignite examples on the cli not IDE, you can export executable Jar file from IDE. Then run it by using below command.nohup java -jar </path/to/your Jar file name>Configuring Ignite InterpreterAt the "Interpreters" menu, you may edit Ignite interpreter or create new one. Zeppelin provides these properties for Ignite. Property Name value Description ignite.addresses 127.0.0.1:47500..47509 Coma separated list of Ignite cluster hosts. See [Ignite Cluster Configuration](https://apacheignite.readme.io/docs/cluster-config) section for more details. ignite.clientMode true You can connect to the Ignite cluster as client or server node. See [Ignite Clients vs. Servers](https://apacheignite.readme.io/docs/clients-vs-servers) section for details. Use true or false values in order to connect in client or server mode respectively. ignite.config.url Configuration URL. Overrides all other settings. ignite.jdbc.url jdbc:ignite:cfg://default-ignite-jdbc.xml Ignite JDBC connection URL. ignite.peerClassLoadingEnabled true Enables peer-class-loading. See [Zero Deployment](https://apacheignite.readme.io/docs/zero-deployment) section for details. Use true or false values in order to enable or disable P2P class loading respectively. How to useAfter configuring Ignite interpreter, create your own notebook. Then you can bind interpreters like below image.For more interpreter binding information see here.Ignite SQL interpreterIn order to execute SQL query, use %ignite.ignitesql prefix. Supposing you are running org.apache.ignite.examples.streaming.wordcount.StreamWords, then you can use "words" cache( Of course you have to specify this cache name to the Ignite interpreter setting section ignite.jdbc.url of Zeppelin ).For example, you can select top 10 words in the words cache using the following query%ignite.ignitesqlselect _val, count(_val) as cnt from String group by _val order by cnt desc limit 10As long as your Ignite version and Zeppelin Ignite version is same, you can also use scala code. Please check the Zeppelin Ignite version before you download your own Ignite.%igniteimport org.apache.ignite._import org.apache.ignite.cache.affinity._import org.apache.ignite.cache.query._import org.apache.ignite.configuration._import scala.collection.JavaConversions._val cache: IgniteCache[AffinityUuid, String] = ignite.cache("words")val qry = new SqlFieldsQuery("select avg(cnt), min(cnt), max(cnt) from (select count(_val) as cnt from String group by _val)", true)val res = cache.query(qry).getAll()collectionAsScalaIterable(res).foreach(println _)Apache Ignite also provides a guide docs for Zeppelin "Ignite with Apache Zeppelin"",
"url": " /interpreter/ignite.html",
"group": "interpreter",
"excerpt": "Apache Ignite in-memory Data Fabric is a high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash technologies."
}
,
"/interpreter/groovy.html": {
"title": "Apache Groovy Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Groovy Interpreter for Apache ZeppelinSamples%groovy//get a parameter defined as z.angularBind('ngSearchParam', value, 'paragraph_id')//g is a context object for groovy to avoid mix with z objectdef param = g.angular('ngSearchParam')//send request https://www.googleapis.com/customsearch/v1?q=ngSearchParam_valuedef r = HTTP.get( //assume you defined the groovy interpreter property // `search_baseurl`='https://www.googleapis.com/customsearch/v1' //in groovy object o.getProperty('A') == o.'A' == o.A == o['A'] url : g.search_baseurl, query: [ q: param ], headers: [ 'Accept':'application/json', //'Authorization:' : g.getProperty('search_auth'), ], ssl : g.getProperty('search_ssl') // assume groovy interpreter property search_ssl = HTTP.getNaiveSSLContext())//check response codeif( r.response.code==200 ) { g.html().with{ //g.html() renders %angular to output and returns groovy.xml.MarkupBuilder h2("the response ${r.response.code}") span( r.response.body ) h2("headers") pre( r.response.headers.join('n') ) }} else { //just to show that it's possible to use println with multiline groovy string to render output println("""%angular <script> alert ("code=${r.response.code} n msg=${r.response.message}") </script> """)}%groovy//renders a table with headers a, b, c and two rowsg.table( [ ['a','b','c'], ['a1','b1','c1'], ['a2','b2','c2'], ])the g objectg.angular(String name)Returns angular object by name. Look up notebook scope first and then global scope.g.angularBind(String name, Object value)Assign a new value into angular object namejava.util.Properties g.getProperties()returns all properties defined for this interpreterString g.getProperty('PROPERTY_NAME') g.PROPERTY_NAMEg.'PROPERTY_NAME'g['PROPERTY_NAME']g.getProperties().getProperty('PROPERTY_NAME')All above the accessor to named property defined in groovy interpreter.In this case with name PROPERTY_NAMEgroovy.xml.MarkupBuilder g.html()Starts or continues rendering of %angular to output and returns groovy.xml.MarkupBuilderMarkupBuilder is usefull to generate html (xml)void g.table(obj)starts or continues rendering table rows.obj: List(rows) of List(columns) where first line is a header g.input(name, value )Creates text input with value specified. The parameter value is optional.g.select(name, default, Map<Object, String> options)Creates select input with defined options. The parameter default is optional.g.select('sex', 'm', ['m':'man', 'w':'woman'])g.checkbox(name, Collection checked, Map<Object, String> options)Creates checkbox input.g.get(name, default)Returns interpreter-based variable. Visibility depends on interpreter scope. The parameter default is optional.g.put(name, value)Stores new value into interpreter-based variable. Visibility depends on interpreter scope.",
"url": " /interpreter/groovy.html",
"group": "interpreter",
"excerpt": "Apache Groovy is a powerful, optionally typed and dynamic language, with static-typing and static compilation capabilities, for the Java platform aimed at improving developer productivity thanks to a concise, familiar and easy to learn syntax."
}
,
"/interpreter/sap.html": {
"title": "SAP BusinessObjects Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->SAP BusinessObjects (Universe) Interpreter for Apache ZeppelinOverviewSAP BusinessObjects BI platform (universes) can simplify the lives of business users and IT staff. SAP BusinessObjects is based on universes. The universe contains dual-semantic layer model. The users make queries upon universes. This interpreter is new interface for universes.Disclaimer SAP interpreter is not official interpreter for SAP BusinessObjects BI platform. It uses BI Semantic Layer REST APIThis interpreter is not directly supported by SAP AG.Tested with versions 4.2SP3 (14.2.3.2220) and 4.2SP5. There is no support for filters in UNX-universes converted from old UNV format.The universe name must be unique.Configuring SAP Universe InterpreterAt the "Interpreters" menu, you can edit SAP interpreter or create new one. Zeppelin provides these properties for SAP. Property Name Value Description universe.api.url http://localhost:6405/biprws The base url for the SAP BusinessObjects BI platform. You have to edit "localhost" that you may use (ex. http://0.0.0.0:6405/biprws) universe.authType secEnterprise The type of authentication for API of Universe. Available values: secEnterprise, secLDAP, secWinAD, secSAPR3 universe.password The BI platform user password universe.user Administrator The BI platform user login How to use Choose the universe Choose dimensions and measures in select statement Define conditions in where statementYou can compare two dimensions/measures or use Filter (without value). Dimesions/Measures can be compared with static values, may be is null or is not null, contains or not in list.Available the nested conditions (using braces "()"). "and" operator have more priority than "or". If generated query contains promtps, then promtps will appear as dynamic form after paragraph submitting.Example query%sapuniverse [Universe Name];select [Folder1].[Dimension2], [Folder2].[Dimension3], [Measure1]where [Filter1] and [Date] > '2018-01-01 00:00:00' and [Folder1].[Dimension4] is not null and [Folder1].[Dimension5] in ('Value1', 'Value2');<<<<<<< HEADdistinct keywordYou can write keyword distinct after keyword select to return only distinct (different) values.Example query```sql%sapuniverse [Universe Name];select distinct [Folder1].[Dimension2], [Measure1]where [Filter1];```limit keywordYou can write keyword limit and limit value in the end of query to limit the number of records returned based on a limit value.Example query```sql%sapuniverse [Universe Name];select [Folder1].[Dimension2], [Measure1]where [Filter1]limit 100;```Object InterpolationThe SAP interpreter also supports interpolation of ZeppelinContext objects into the paragraph text.To enable this feature set universe.interpolation to true. The following example shows one use of this facility:In Scala cell:z.put("curr_date", "2018-01-01 00:00:00")In later SAP cell:where [Filter1] and [Date] > '{curr_date}'",
"url": " /interpreter/sap.html",
"group": "interpreter",
"excerpt": "SAP BusinessObjects BI platform can simplify the lives of business users and IT staff. SAP BusinessObjects is based on universes. The universe contains dual-semantic layer model. The users make queries upon universes. This interpreter is new interface for universes."
}
,
"/interpreter/kylin.html": {
"title": "Apache Kylin Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Kylin Interpreter for Apache ZeppelinOverviewApache Kylin is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets, original contributed from eBay Inc. The interpreter assumes that Apache Kylin has been installed and you can connect to Apache Kylin from the machine Apache Zeppelin is installed.To get start with Apache Kylin, please see Apache Kylin Quickstart.Configuration Name Default Description kylin.api.url http://localhost:7070/kylin/api/query kylin query POST API The format can be like http://<host>:<port>/kylin/api/query kylin.api.user ADMIN kylin user kylin.api.password KYLIN kylin password kylin.query.project learn_kylin String, Project to perform query. Could update at notebook level kylin.query.ispartial true true|false (@Deprecated since Apache Kylin V1.5) Whether accept a partial result or not, default be “false”. Set to “false” for production use. kylin.query.limit 5000 int, Query limit If limit is set in sql, perPage will be ignored. kylin.query.offset 0 int, Query offset If offset is set in sql, curIndex will be ignored. Using the Apache Kylin InterpreterIn a paragraph, use %kylin(project_name) to select the kylin interpreter, project name and then input sql. If no project name defined, will use the default project name from the above configuration.%kylin(learn_project)select count(*) from kylin_sales group by part_dt",
"url": " /interpreter/kylin.html",
"group": "interpreter",
"excerpt": "Apache Kylin™ is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets, original contributed from eBay Inc. ."
}
,
"/interpreter/hdfs.html": {
"title": "HDFS File System Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->HDFS File System Interpreter for Apache ZeppelinOverviewHadoop File System is a distributed, fault tolerant file system part of the hadoop project and is often used as storage for distributed processing engines like Hadoop MapReduce and Apache Spark or underlying file systems like Alluxio.Configuration Property Default Description hdfs.url http://localhost:50070/webhdfs/v1/ The URL for WebHDFS hdfs.user hdfs The WebHDFS user hdfs.maxlength 1000 Maximum number of lines of results fetched This interpreter connects to HDFS using the HTTP WebHDFS interface.It supports the basic shell file commands applied to HDFS, it currently only supports browsing.You can use ls [PATH] and ls -l [PATH] to list a directory. If the path is missing, then the current directory is listed. ls supports a -h flag for human readable file sizes.You can use cd [PATH] to change your current directory by giving a relative or an absolute path.You can invoke pwd to see your current directory.Tip : Use ( Ctrl + . ) for autocompletion.Create InterpreterIn a notebook, to enable the HDFS interpreter, click the Gear icon and select HDFS.WebHDFS REST APIYou can confirm that you're able to access the WebHDFS API by running a curl command against the WebHDFS end point provided to the interpreter.Here is an example:$> curl "http://localhost:50070/webhdfs/v1/?op=LISTSTATUS"",
"url": " /interpreter/hdfs.html",
"group": "interpreter",
"excerpt": "Hadoop File System is a distributed, fault tolerant file system part of the hadoop project and is often used as storage for distributed processing engines like Hadoop MapReduce and Apache Spark or underlying file systems like Alluxio."
}
,
"/interpreter/hbase.html": {
"title": "HBase Shell Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->HBase Shell Interpreter for Apache ZeppelinOverviewHBase Shell is a JRuby IRB client for Apache HBase. This interpreter provides all capabilities of Apache HBase shell within Apache Zeppelin. The interpreter assumes that Apache HBase client software has been installed and it can connect to the Apache HBase cluster from the machine on where Apache Zeppelin is installed.To get start with HBase, please see HBase Quickstart.HBase release supportedBy default, Zeppelin is built against HBase 1.0.x releases. To work with HBase 1.1.x releases, use the following build command:# HBase 1.1.4mvn clean package -DskipTests -Phadoop-2.6 -Dhadoop.version=2.6.0 -P build-distr -Dhbase.hbase.version=1.1.4 -Dhbase.hadoop.version=2.6.0To work with HBase 1.2.0+, use the following build command:# HBase 1.2.0mvn clean package -DskipTests -Phadoop-2.6 -Dhadoop.version=2.6.0 -P build-distr -Dhbase.hbase.version=1.2.0 -Dhbase.hadoop.version=2.6.0Configuration Property Default Description hbase.home /usr/lib/hbase Installation directory of HBase, defaults to HBASE_HOME in environment hbase.ruby.sources lib/ruby Path to Ruby scripts relative to 'hbase.home' zeppelin.hbase.test.mode false Disable checks for unit and manual tests If you want to connect to HBase running on a cluster, you'll need to follow the next step.Export HBASE_HOMEIn conf/zeppelin-env.sh, export HBASE_HOME environment variable with your HBase installation path. This ensures hbase-site.xml can be loaded.For exampleexport HBASE_HOME=/usr/lib/hbaseor, when running with CDHexport HBASE_HOME="/opt/cloudera/parcels/CDH/lib/hbase"You can optionally export HBASE_CONF_DIR instead of HBASE_HOME should you have custom HBase configurations.Enabling the HBase Shell InterpreterIn a notebook, to enable the HBase Shell interpreter, click the Gear icon and select HBase Shell.Using the HBase Shell InterpreterIn a paragraph, use %hbase to select the HBase Shell interpreter and then input all commands. To get the list of available commands, use help.%hbasehelpFor example, to create a table%hbasecreate 'test', 'cf'And then to put data into that table%hbaseput 'test', 'row1', 'cf:a', 'value1'For more information on all commands available, refer to HBase shell commands.",
"url": " /interpreter/hbase.html",
"group": "interpreter",
"excerpt": "HBase Shell is a JRuby IRB client for Apache HBase. This interpreter provides all capabilities of Apache HBase shell within Apache Zeppelin."
}
,
"/interpreter/beam.html": {
"title": "Beam interpreter in Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Beam interpreter for Apache ZeppelinOverviewApache Beam is an open source unified platform for data processing pipelines. A pipeline can be build using one of the Beam SDKs.The execution of the pipeline is done by different Runners. Currently, Beam supports Apache Flink Runner, Apache Spark Runner, and Google Dataflow Runner.How to useBasically, you can write normal Beam java code where you can determine the Runner. You should write the main method inside a class becuase the interpreter invoke this main to execute the pipeline. Unlike Zeppelin normal pattern, each paragraph is considered as a separate job, there isn't any relation to any other paragraph.The following is a demonstration of a word count example with data represented in array of stringsBut it can read data from files by replacing Create.of(SENTENCES).withCoder(StringUtf8Coder.of()) with TextIO.Read.from("path/to/filename.txt")%beam// most used importsimport org.apache.beam.sdk.coders.StringUtf8Coder;import org.apache.beam.sdk.transforms.Create;import java.io.Serializable;import java.util.Arrays;import java.util.List;import java.util.ArrayList;import org.apache.beam.runners.direct.*;import org.apache.beam.sdk.runners.*;import org.apache.beam.sdk.options.*;import org.apache.beam.runners.flink.*;import org.apache.beam.sdk.Pipeline;import org.apache.beam.sdk.io.TextIO;import org.apache.beam.sdk.options.PipelineOptionsFactory;import org.apache.beam.sdk.transforms.Count;import org.apache.beam.sdk.transforms.DoFn;import org.apache.beam.sdk.transforms.MapElements;import org.apache.beam.sdk.transforms.ParDo;import org.apache.beam.sdk.transforms.SimpleFunction;import org.apache.beam.sdk.values.KV;import org.apache.beam.sdk.options.PipelineOptions;public class MinimalWordCount { static List<String> s = new ArrayList<>(); static final String[] SENTENCES_ARRAY = new String[] { "Hadoop is the Elephant King!", "A yellow and elegant thing.", "He never forgets", "Useful data, or lets", "An extraneous element cling!", "A wonderful king is Hadoop.", "The elephant plays well with Sqoop.", "But what helps him to thrive", "Are Impala, and Hive,", "And HDFS in the group.", "Hadoop is an elegant fellow.", "An elephant gentle and mellow.", "He never gets mad,", "Or does anything bad,", "Because, at his core, he is yellow", }; static final List<String> SENTENCES = Arrays.asList(SENTENCES_ARRAY); public static void main(String[] args) { PipelineOptions options = PipelineOptionsFactory.create().as(PipelineOptions.class); options.setRunner(FlinkRunner.class); Pipeline p = Pipeline.create(options); p.apply(Create.of(SENTENCES).withCoder(StringUtf8Coder.of())) .apply("ExtractWords", ParDo.of(new DoFn<String, String>() { @ProcessElement public void processElement(ProcessContext c) { for (String word : c.element().split("[^a-zA-Z']+")) { if (!word.isEmpty()) { c.output(word); } } } })) .apply(Count.<String> perElement()) .apply("FormatResults", ParDo.of(new DoFn<KV<String, Long>, String>() { @ProcessElement public void processElement(DoFn<KV<String, Long>, String>.ProcessContext arg0) throws Exception { s.add("n" + arg0.element().getKey() + "t" + arg0.element().getValue()); } })); p.run(); System.out.println("%table wordtcount"); for (int i = 0; i < s.size(); i++) { System.out.print(s.get(i)); } }}",
"url": " /interpreter/beam.html",
"group": "interpreter",
"excerpt": "Apache Beam is an open source, unified programming model that you can use to create a data processing pipeline."
}
,
"/interpreter/geode.html": {
"title": "Geode/Gemfire OQL Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Geode/Gemfire OQL Interpreter for Apache ZeppelinOverview Name Class Description %geode.oql GeodeOqlInterpreter Provides OQL environment for Apache Geode This interpreter supports the Geode Object Query Language (OQL).With the OQL-based querying language:You can query on any arbitrary objectYou can navigate object collectionsYou can invoke methods and access the behavior of objectsData mapping is supportedYou are not required to declare types. Since you do not need type definitions, you can work across multiple languagesYou are not constrained by a schemaThis Video Tutorial illustrates some of the features provided by the Geode Interpreter.Create InterpreterBy default Zeppelin creates one Geode/OQL instance. You can remove it or create more instances.Multiple Geode instances can be created, each configured to the same or different backend Geode cluster. But over time a Notebook can have only one Geode interpreter instance bound. That means you cannot connect to different Geode clusters in the same Notebook. This is a known Zeppelin limitation.To create new Geode instance open the Interpreter section and click the +Create button. Pick a Name of your choice and from the Interpreter drop-down select geode.Then follow the configuration instructions and Save the new instance.Note: The Name of the instance is used only to distinguish the instances while binding them to the Notebook. The Name is irrelevant inside the Notebook. In the Notebook you must use %geode.oql tag.Bind to NotebookIn the Notebook click on the settings icon in the top right corner. The select/deselect the interpreters to be bound with the Notebook.ConfigurationYou can modify the configuration of the Geode from the Interpreter section. The Geode interpreter expresses the following properties: Property Name Description Default Value geode.locator.host The Geode Locator Host localhost geode.locator.port The Geode Locator Port 10334 geode.max.result Max number of OQL result to display to prevent the browser overload 1000 How to useTip 1: Use (CTRL + .) for OQL auto-completion.Tip 2: Always start the paragraphs with the full %geode.oql prefix tag! The short notation: %geode would still be able run the OQL queries but the syntax highlighting and the auto-completions will be disabled.Create / Destroy RegionsThe OQL specification does not support Geode Regions mutation operations. To create/destroy regions one should use the GFSH shell tool instead. In the following it is assumed that the GFSH is colocated with Zeppelin server.%shsource /etc/geode/conf/geode-env.shgfsh << EOF connect --locator=ambari.localdomain[10334] destroy region --name=/regionEmployee destroy region --name=/regionCompany create region --name=regionEmployee --type=REPLICATE create region --name=regionCompany --type=REPLICATE exit;EOFAbove snippet re-creates two regions: regionEmployee and regionCompany. Note that you have to explicitly specify the locator host and port. The values should match those you have used in the Geode Interpreter configuration. Comprehensive list of GFSH Commands by Functional Area.Basic OQL%geode.oqlSELECT count(*) FROM /regionEmployeeOQL IN and SET filters:%geode.oqlSELECT * FROM /regionEmployeeWHERE companyId IN SET(2) OR lastName IN SET('Tzolov13', 'Tzolov73')OQL JOIN operations%geode.oqlSELECT e.employeeId, e.firstName, e.lastName, c.id as companyId, c.companyName, c.addressFROM /regionEmployee e, /regionCompany cWHERE e.companyId = c.idBy default the QOL responses contain only the region entry values. To access the keys, query the EntrySet instead:%geode.oqlSELECT e.key, e.value.companyId, e.value.emailFROM /regionEmployee.entrySet eFollowing query will return the EntrySet value as a Blob:%geode.oqlSELECT e.key, e.value FROM /regionEmployee.entrySet eNote: You can have multiple queries in the same paragraph but only the result from the first is displayed. [1], [2].GFSH Commands From The ShellUse the Shell Interpreter (%sh) to run OQL commands form the command line:%shsource /etc/geode/conf/geode-env.shgfsh -e "connect" -e "list members"Apply Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form inside your OQL queries. You can use both the text input and select form parameterization features%geode.oqlSELECT * FROM /regionEmployee e WHERE e.employeeId > ${Id}Auto-completionThe Geode Interpreter provides a basic auto-completion functionality. On (Ctrl+.) it list the most relevant suggestions in a pop-up window.Geode REST APITo list the defined regions you can use the Geode REST API:http://<geode server hostname>phd1.localdomain:8484/gemfire-api/v1/{ "regions" : [{ "name" : "regionEmployee", "type" : "REPLICATE", "key-constraint" : null, "value-constraint" : null }, { "name" : "regionCompany", "type" : "REPLICATE", "key-constraint" : null, "value-constraint" : null }]}To enable Geode REST API with JSON support add the following properties to geode.server.properties.file and restart:http-service-port=8484start-dev-rest-api=true",
"url": " /interpreter/geode.html",
"group": "interpreter",
"excerpt": "Apache Geode (incubating) provides a database-like consistency model, reliable transaction processing and a shared-nothing architecture to maintain very low latency performance with high concurrency processing."
}
,
"/interpreter/r.html": {
"title": "R Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->R Interpreter for Apache ZeppelinOverviewR is a free software environment for statistical computing and graphics.To run R code and visualize plots in Apache Zeppelin, you will need R on your master node (or your dev laptop).For Centos: yum install R R-devel libcurl-devel openssl-develFor Ubuntu: apt-get install r-baseValidate your installation with a simple R command:R -e "print(1+1)"To enjoy plots, install additional libraries with:devtools with R -e "install.packages('devtools', repos = 'http://cran.us.r-project.org')"knitr with R -e "install.packages('knitr', repos = 'http://cran.us.r-project.org')"ggplot2 withR -e "install.packages('ggplot2', repos = 'http://cran.us.r-project.org')"Other visualization libraries: R -e "install.packages(c('devtools','mplot', 'googleVis'), repos = 'http://cran.us.r-project.org'); require(devtools); install_github('ramnathv/rCharts')"We recommend you to also install the following optional R libraries for happy data analytics:glmnetpROCdata.tablecaretsqldfwordcloudConfigurationTo run Zeppelin with the R Interpreter, the SPARK_HOME environment variable must be set. The best way to do this is by editing conf/zeppelin-env.sh.If it is not set, the R Interpreter will not be able to interface with Spark.You should also copy conf/zeppelin-site.xml.template to conf/zeppelin-site.xml. That will ensure that Zeppelin sees the R Interpreter the first time it starts up.Using the R InterpreterBy default, the R Interpreter appears as two Zeppelin Interpreters, %r and %knitr.%r will behave like an ordinary REPL. You can execute commands as in the CLI. R base plotting is fully supportedIf you return a data.frame, Zeppelin will attempt to display it using Zeppelin's built-in visualizations.%knitr interfaces directly against knitr, with chunk options on the first line:The two interpreters share the same environment. If you define a variable from %r, it will be within-scope if you then make a call using knitr.Using SparkR & Moving Between LanguagesIf SPARK_HOME is set, the SparkR package will be loaded automatically:The Spark Context and SQL Context are created and injected into the local environment automatically as sc and sql.The same context are shared with the %spark, %sql and %pyspark interpreters:You can also make an ordinary R variable accessible in scala and Python:And vice versa:Caveats & TroubleshootingAlmost all issues with the R interpreter turned out to be caused by an incorrectly set SPARK_HOME. The R interpreter must load a version of the SparkR package that matches the running version of Spark, and it does this by searching SPARK_HOME. If Zeppelin isn't configured to interface with Spark in SPARK_HOME, the R interpreter will not be able to connect to Spark.The knitr environment is persistent. If you run a chunk from Zeppelin that changes a variable, then run the same chunk again, the variable has already been changed. Use immutable variables.(Note that %spark.r and %r are two different ways of calling the same interpreter, as are %spark.knitr and %knitr. By default, Zeppelin puts the R interpreters in the %spark. Interpreter Group.Using the %r interpreter, if you return a data.frame, HTML, or an image, it will dominate the result. So if you execute three commands, and one is hist(), all you will see is the histogram, not the results of the other commands. This is a Zeppelin limitation.If you return a data.frame (for instance, from calling head()) from the %spark.r interpreter, it will be parsed by Zeppelin's built-in data visualization system. Why knitr Instead of rmarkdown? Why no htmlwidgets? In order to support htmlwidgets, which has indirect dependencies, rmarkdown uses pandoc, which requires writing to and reading from disc. This makes it many times slower than knitr, which can operate entirely in RAM.Why no ggvis or shiny? Supporting shiny would require integrating a reverse-proxy into Zeppelin, which is a task.Max OS X & case-insensitive filesystem. If you try to install on a case-insensitive filesystem, which is the Mac OS X default, maven can unintentionally delete the install directory because r and R become the same subdirectory.Error unable to start device X11 with the repl interpreter. Check your shell login scripts to see if they are adjusting the DISPLAY environment variable. This is common on some operating systems as a workaround for ssh issues, but can interfere with R plotting.akka Library Version or TTransport errors. This can happen if you try to run Zeppelin with a SPARK_HOME that has a version of Spark other than the one specified with -Pspark-1.x when Zeppelin was compiled.",
"url": " /interpreter/r.html",
"group": "interpreter",
"excerpt": "R is a free software environment for statistical computing and graphics."
}
,
"/interpreter/flink.html": {
"title": "Flink Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Flink interpreter for Apache ZeppelinOverviewApache Flink is an open source platform for distributed stream and batch data processing. Flink’s core is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams. Flink also builds batch processing on top of the streaming engine, overlaying native iteration support, managed memory, and program optimization.How to start local Flink cluster, to test the interpreterZeppelin comes with pre-configured flink-local interpreter, which starts Flink in a local mode on your machine, so you do not need to install anything.How to configure interpreter to point to Flink clusterAt the "Interpreters" menu, you have to create a new Flink interpreter and provide next properties: property value Description host local host name of running JobManager. 'local' runs flink in local mode (default) port 6123 port of running JobManager For more information about Flink configuration, you can find it here.How to test it's workingYou can find an example of Flink usage in the Zeppelin Tutorial folder or try the following word count example, by using the Zeppelin notebook from Till Rohrmann's presentation Interactive data analysis with Apache Flink for Apache Flink Meetup.%shrm 10.txt.utf-8wget http://www.gutenberg.org/ebooks/10.txt.utf-8%flinkcase class WordCount(word: String, frequency: Int)val bible:DataSet[String] = benv.readTextFile("10.txt.utf-8")val partialCounts: DataSet[WordCount] = bible.flatMap{ line => """bw+b""".r.findAllIn(line).map(word => WordCount(word, 1))// line.split(" ").map(word => WordCount(word, 1))}val wordCounts = partialCounts.groupBy("word").reduce{ (left, right) => WordCount(left.word, left.frequency + right.frequency)}val result10 = wordCounts.first(10).collect()",
"url": " /interpreter/flink.html",
"group": "interpreter",
"excerpt": "Apache Flink is an open source platform for distributed stream and batch data processing."
}
,
"/interpreter/postgresql.html": {
"title": "PostgreSQL, Apache HAWQ (incubating) Interpreter for Apache Zeppelin",
"content" : "<!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> PostgreSQL, Apache HAWQ (incubating) Interpreter for Apache Zeppelin Important NoticePostgresql interpreter is deprecated and merged into JDBC Interpreter. You can use it with JDBC Interpreter as same functionality. See Postgresql setting example for more detailed information.",
"url": " /interpreter/postgresql.html",
"group": "interpreter",
"excerpt": "Apache Zeppelin supports PostgreSQL, Apache HAWQ(incubating) and Greenplum SQL data processing engines."
}
,
"/interpreter/cassandra.html": {
"title": "Cassandra CQL Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Cassandra CQL Interpreter for Apache Zeppelin Name Class Description %cassandra CassandraInterpreter Provides interpreter for Apache Cassandra CQL query language Enabling Cassandra InterpreterIn a notebook, to enable the Cassandra interpreter, click on the Gear icon and select Cassandra Using the Cassandra InterpreterIn a paragraph, use %cassandra to select the Cassandra interpreter and then input all commands.To access the interactive help, type HELP; Interpreter CommandsThe Cassandra interpreter accepts the following commands Command Type Command Name Description Help command HELP Display the interactive help menu Schema commands DESCRIBE KEYSPACE, DESCRIBE CLUSTER, DESCRIBE TABLES ... Custom commands to describe the Cassandra schema Option commands @consistency, @retryPolicy, @fetchSize ... Inject runtime options to all statements in the paragraph Prepared statement commands @prepare, @bind, @remove_prepared Let you register a prepared command and re-use it later by injecting bound values Native CQL statements All CQL-compatible statements (SELECT, INSERT, CREATE, ...) All CQL statements are executed directly against the Cassandra server CQL statementsThis interpreter is compatible with any CQL statement supported by Cassandra. Ex:INSERT INTO users(login,name) VALUES('jdoe','John DOE');SELECT * FROM users WHERE login='jdoe';Each statement should be separated by a semi-colon ( ; ) except the special commands below:@prepare@bind@remove_prepare@consistency@serialConsistency@timestamp@retryPolicy@fetchSize@requestTimeOutMulti-line statements as well as multiple statements on the same line are also supported as long as they are separated by a semi-colon. Ex:USE spark_demo;SELECT * FROM albums_by_country LIMIT 1; SELECT * FROM countries LIMIT 1;SELECT *FROM artistsWHERE login='jlennon';Batch statements are supported and can span multiple lines, as well as DDL (CREATE/ALTER/DROP) statements:BEGIN BATCH INSERT INTO users(login,name) VALUES('jdoe','John DOE'); INSERT INTO users_preferences(login,account_type) VALUES('jdoe','BASIC');APPLY BATCH;CREATE TABLE IF NOT EXISTS test( key int PRIMARY KEY, value text);CQL statements are case-insensitive (except for column names and values). This means that the following statements are equivalent and valid:INSERT INTO users(login,name) VALUES('jdoe','John DOE');Insert into users(login,name) vAlues('hsue','Helen SUE');The complete list of all CQL statements and versions can be found below: Cassandra Version Documentation Link 3.x <a target="_blank" href="http://docs.datastax.com/en/cql/3.3/cql/cqlIntro.html"> http://docs.datastax.com/en/cql/3.3/cql/cqlIntro.html 2.2 <a target="_blank" href="http://docs.datastax.com/en/cql/3.3/cql/cqlIntro.html"> http://docs.datastax.com/en/cql/3.3/cql/cqlIntro.html 2.1 & 2.0 <a target="_blank" href="http://docs.datastax.com/en/cql/3.1/cql/cql_intro_c.html"> http://docs.datastax.com/en/cql/3.1/cql/cqlintroc.html 1.2 <a target="_blank" href="http://docs.datastax.com/en/cql/3.0/cql/aboutCQL.html"> http://docs.datastax.com/en/cql/3.0/cql/aboutCQL.html Comments in statementsIt is possible to add comments between statements. Single line comments start with the hash sign (#) or double slashes (//). Multi-line comments are enclosed between /** and **/. Ex:#Single line comment style 1INSERT INTO users(login,name) VALUES('jdoe','John DOE');//Single line comment style 2/** Multi line comments **/Insert into users(login,name) vAlues('hsue','Helen SUE');Syntax ValidationThe interpreters is shipped with a built-in syntax validator. This validator only checks for basic syntax errors.All CQL-related syntax validation is delegated directly to CassandraMost of the time, syntax errors are due to missing semi-colons between statements or typo errors.Schema commandsTo make schema discovery easier and more interactive, the following commands are supported: Command Description DESCRIBE CLUSTER; Show the current cluster name and its partitioner DESCRIBE KEYSPACES; List all existing keyspaces in the cluster and their configuration (replication factor, durable write ...) DESCRIBE TABLES; List all existing keyspaces in the cluster and for each, all the tables name DESCRIBE TYPES; List all existing keyspaces in the cluster and for each, all the user-defined types name DESCRIBE FUNCTIONS; List all existing keyspaces in the cluster and for each, all the functions name DESCRIBE AGGREGATES; List all existing keyspaces in the cluster and for each, all the aggregates name DESCRIBE MATERIALIZED VIEWS; List all existing keyspaces in the cluster and for each, all the materialized views name DESCRIBE KEYSPACE <keyspacename>; Describe the given keyspace configuration and all its table details (name, columns, ...) DESCRIBE TABLE (<keyspacename>).<tablename>; Describe the given table. If the keyspace is not provided, the current logged in keyspace is used. If there is no logged in keyspace, the default system keyspace is used. If no table is found, an error message is raised DESCRIBE TYPE (<keyspacename>).<typename>; Describe the given type(UDT). If the keyspace is not provided, the current logged in keyspace is used. If there is no logged in keyspace, the default system keyspace is used. If no type is found, an error message is raised DESCRIBE FUNCTION (<keyspacename>).<functionname>; Describe the given function. If the keyspace is not provided, the current logged in keyspace is used. If there is no logged in keyspace, the default system keyspace is used. If no function is found, an error message is raised DESCRIBE AGGREGATE (<keyspacename>).<aggregatename>; Describe the given aggregate. If the keyspace is not provided, the current logged in keyspace is used. If there is no logged in keyspace, the default system keyspace is used. If no aggregate is found, an error message is raised DESCRIBE MATERIALIZED VIEW (<keyspacename>).<view_name>; Describe the given view. If the keyspace is not provided, the current logged in keyspace is used. If there is no logged in keyspace, the default system keyspace is used. If no view is found, an error message is raised The schema objects (cluster, keyspace, table, type, function and aggregate) are displayed in a tabular format.There is a drop-down menu on the top left corner to expand objects details. On the top right menu is shown the Icon legend. Runtime ParametersSometimes you want to be able to pass runtime query parameters to your statements.Those parameters are not part of the CQL specs and are specific to the interpreter.Below is the list of all parameters: Parameter Syntax Description Consistency Level @consistency=value Apply the given consistency level to all queries in the paragraph Serial Consistency Level @serialConsistency=value Apply the given serial consistency level to all queries in the paragraph Timestamp @timestamp=long value Apply the given timestamp to all queries in the paragraph. Please note that timestamp value passed directly in CQL statement will override this value Retry Policy @retryPolicy=value Apply the given retry policy to all queries in the paragraph Fetch Size @fetchSize=integer value Apply the given fetch size to all queries in the paragraph Request Time Out @requestTimeOut=integer value Apply the given request timeout in millisecs to all queries in the paragraph Some parameters only accept restricted values: Parameter Possible Values Consistency Level ALL, ANY, ONE, TWO, THREE, QUORUM, LOCALONE, LOCALQUORUM, EACHQUORUM Serial Consistency Level SERIAL, LOCALSERIAL Timestamp Any long value Retry Policy DEFAULT, DOWNGRADINGCONSISTENCY, FALLTHROUGH, LOGGINGDEFAULT, LOGGINGDOWNGRADING, LOGGINGFALLTHROUGH Fetch Size Any integer value Please note that you should not add semi-colon ( ; ) at the end of each parameter statementSome examples:CREATE TABLE IF NOT EXISTS spark_demo.ts( key int PRIMARY KEY, value text);TRUNCATE spark_demo.ts;// Timestamp in the past@timestamp=10// Force timestamp directly in the first insertINSERT INTO spark_demo.ts(key,value) VALUES(1,'first insert') USING TIMESTAMP 100;// Select some data to make the clock turnSELECT * FROM spark_demo.albums LIMIT 100;// Now insert using the timestamp parameter set at the beginning(10)INSERT INTO spark_demo.ts(key,value) VALUES(1,'second insert');// Check for the result. You should see 'first insert'SELECT value FROM spark_demo.ts WHERE key=1;Some remarks about query parameters:many query parameters can be set in the same paragraphif the same query parameter is set many time with different values, the interpreter only take into account the first valueeach query parameter applies to all CQL statements in the same paragraph, unless you override the option using plain CQL text (like forcing timestamp with the USING clause)the order of each query parameter with regard to CQL statement does not matterSupport for Prepared StatementsFor performance reason, it is better to prepare statements before-hand and reuse them later by providing bound values.This interpreter provides 3 commands to handle prepared and bound statements:@prepare@bind@remove_preparedExample:@prepare[statement-name]=...@bind[statement-name]=’text’, 1223, ’2015-07-30 12:00:01’, null, true, [‘list_item1’, ’list_item2’]@bind[statement-name-with-no-bound-value]@remove_prepare[statement-name]@prepareYou can use the syntax "@prepare[statement-name]=SELECT..." to create a prepared statement.The statement-name is mandatory because the interpreter prepares the given statement with the Java driver andsaves the generated prepared statement in an internal hash map, using the provided statement-name as search key.Please note that this internal prepared statement map is shared with all notebooks and all paragraphs becausethere is only one instance of the interpreter for CassandraIf the interpreter encounters many @prepare for the same statement-name (key), only the first statement will be taken into account.Example:@prepare[select]=SELECT * FROM spark_demo.albums LIMIT ?@prepare[select]=SELECT * FROM spark_demo.artists LIMIT ?For the above example, the prepared statement is SELECT * FROM spark_demo.albums LIMIT ?.SELECT * FROM spark_demo.artists LIMIT ? is ignored because an entry already exists in the prepared statements map with the key select.In the context of Zeppelin, a notebook can be scheduled to be executed at regular interval,thus it is necessary to avoid re-preparing many time the same statement (considered an anti-pattern).@bindOnce the statement is prepared (possibly in a separated notebook/paragraph). You can bind values to it:@bind[select_first]=10Bound values are not mandatory for the @bind statement. However if you provide bound values, they need to comply to some syntax:String values should be enclosed between simple quotes (')Date values should be enclosed between simple quotes (') and respect the formats (full list is in the documentation):yyyy-MM-dd HH:MM:ssyyyy-MM-dd HH:MM:ss.SSSnull is parsed as-isboolean (true|false) are parsed as-iscollection values must follow the standard CQL syntax:list: ['listitem1', 'listitem2', ...]set: {'setitem1', 'setitem2', …}map: {'key1': 'val1', 'key2': 'val2', …}tuple values should be enclosed between parenthesis (see Tuple CQL syntax): ('text', 123, true)udt values should be enclosed between brackets (see UDT CQL syntax): {streename: 'Beverly Hills', number: 104, zipcode: 90020, state: 'California', …}It is possible to use the @bind statement inside a batch:BEGIN BATCH @bind[insert_user]='jdoe','John DOE' UPDATE users SET age = 27 WHERE login='hsue';APPLY BATCH;@remove_prepareTo avoid for a prepared statement to stay forever in the prepared statement map, you can use the@remove_prepare[statement-name] syntax to remove it.Removing a non-existing prepared statement yields no error.Using Dynamic FormsInstead of hard-coding your CQL queries, it is possible to use [Zeppelin dynamic form] syntax to inject simple value or multiple choices forms.The legacy mustache syntax ( {{ }} ) to bind input text and select form is still supported but is deprecated and will be removed in future releases.LegacyThe syntax for simple parameter is: {{input_Label=default value}}. The default value is mandatory because the first time the paragraph is executed,we launch the CQL query before rendering the form so at least one value should be provided.The syntax for multiple choices parameter is: {{input_Label=value1 | value2 | … | valueN }}. By default the first choice is used for CQL querythe first time the paragraph is executed.Example:#Secondary index on performer styleSELECT name, country, performerFROM spark_demo.performersWHERE name='${performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}'AND styles CONTAINS '${style=Rock}';In the above example, the first CQL query will be executed for performer='Sheryl Crow' AND style='Rock'.For subsequent queries, you can change the value directly using the form.Please note that we enclosed the ${ } block between simple quotes ( ' ) because Cassandra expects a String here.We could have also use the ${style='Rock'} syntax but this time, the value displayed on the form is 'Rock' and not Rock.It is also possible to use dynamic forms for prepared statements:@bind[select]=='${performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}', '${style=Rock}'Shared statesIt is possible to execute many paragraphs in parallel. However, at the back-end side, we're still using synchronous queries.Asynchronous execution is only possible when it is possible to return a Future value in the InterpreterResult.It may be an interesting proposal for the Zeppelin project.Recently, Zeppelin allows you to choose the level of isolation for your interpreters (see [Interpreter Binding Mode] ).Long story short, you have 3 available bindings:shared : same JVM and same Interpreter instance for all notesscoped : same JVM but different Interpreter instances, one for each noteisolated: different JVM running a single Interpreter instance, one JVM for each noteUsing the shared binding, the same com.datastax.driver.core.Session object is used for all notes and paragraphs.Consequently, if you use the USE keyspace_name; statement to log into a keyspace, it will change the keyspace forall current users of the Cassandra interpreter because we only create 1 com.datastax.driver.core.Session objectper instance of Cassandra interpreter.The same remark does apply to the prepared statement hash map, it is shared by all users using the same instance of Cassandra interpreter.When using scoped binding, in the same JVM Zeppelin will create multiple instances of the Cassandra interpreter, thus multiple com.datastax.driver.core.Session objects. Beware of resource and memory usage using this binding ! The isolated mode is the most extreme and will create as many JVM/com.datastax.driver.core.Session object as there are distinct notes.Interpreter ConfigurationTo configure the Cassandra interpreter, go to the Interpreter menu and scroll down to change the parameters.The Cassandra interpreter is using the official Cassandra Java Driver and most of the parameters are usedto configure the Java driverBelow are the configuration parameters and their default values. Property Name Description Default Value cassandra.cluster Name of the Cassandra cluster to connect to Test Cluster cassandra.compression.protocol On wire compression. Possible values are: NONE, SNAPPY, LZ4 NONE cassandra.credentials.username If security is enable, provide the login none cassandra.credentials.password If security is enable, provide the password none cassandra.hosts Comma separated Cassandra hosts (DNS name or IP address). Ex: 192.168.0.12,node2,node3 localhost cassandra.interpreter.parallelism Number of concurrent paragraphs(queries block) that can be executed 10 cassandra.keyspace Default keyspace to connect to. It is strongly recommended to let the default value and prefix the table name with the actual keyspace in all of your queries system cassandra.load.balancing.policy Load balancing policy. Default = new TokenAwarePolicy(new DCAwareRoundRobinPolicy()) To Specify your own policy, provide the fully qualify class name (FQCN) of your policy. At runtime the interpreter will instantiate the policy using Class.forName(FQCN) DEFAULT cassandra.max.schema.agreement.wait.second Cassandra max schema agreement wait in second 10 cassandra.pooling.core.connection.per.host.local Protocol V2 and below default = 2. Protocol V3 and above default = 1 2 cassandra.pooling.core.connection.per.host.remote Protocol V2 and below default = 1. Protocol V3 and above default = 1 1 cassandra.pooling.heartbeat.interval.seconds Cassandra pool heartbeat interval in secs 30 cassandra.pooling.idle.timeout.seconds Cassandra idle time out in seconds 120 cassandra.pooling.max.connection.per.host.local Protocol V2 and below default = 8. Protocol V3 and above default = 1 8 cassandra.pooling.max.connection.per.host.remote Protocol V2 and below default = 2. Protocol V3 and above default = 1 2 cassandra.pooling.max.request.per.connection.local Protocol V2 and below default = 128. Protocol V3 and above default = 1024 128 cassandra.pooling.max.request.per.connection.remote Protocol V2 and below default = 128. Protocol V3 and above default = 256 128 cassandra.pooling.new.connection.threshold.local Protocol V2 and below default = 100. Protocol V3 and above default = 800 100 cassandra.pooling.new.connection.threshold.remote Protocol V2 and below default = 100. Protocol V3 and above default = 200 100 cassandra.pooling.pool.timeout.millisecs Cassandra pool time out in millisecs 5000 cassandra.protocol.version Cassandra binary protocol version 4 cassandra.query.default.consistency Cassandra query default consistency level Available values: ONE, TWO, THREE, QUORUM, LOCAL_ONE, LOCAL_QUORUM, EACH_QUORUM, ALL ONE cassandra.query.default.fetchSize Cassandra query default fetch size 5000 cassandra.query.default.serial.consistency Cassandra query default serial consistency level Available values: SERIAL, LOCAL_SERIAL SERIAL cassandra.reconnection.policy Cassandra Reconnection Policy. Default = new ExponentialReconnectionPolicy(1000, 10 * 60 * 1000) To Specify your own policy, provide the fully qualify class name (FQCN) of your policy. At runtime the interpreter will instantiate the policy using Class.forName(FQCN) DEFAULT cassandra.retry.policy Cassandra Retry Policy. Default = DefaultRetryPolicy.INSTANCE To Specify your own policy, provide the fully qualify class name (FQCN) of your policy. At runtime the interpreter will instantiate the policy using Class.forName(FQCN) DEFAULT cassandra.socket.connection.timeout.millisecs Cassandra socket default connection timeout in millisecs 500 cassandra.socket.read.timeout.millisecs Cassandra socket read timeout in millisecs 12000 cassandra.socket.tcp.no_delay Cassandra socket TCP no delay true cassandra.speculative.execution.policy Cassandra Speculative Execution Policy. Default = NoSpeculativeExecutionPolicy.INSTANCE To Specify your own policy, provide the fully qualify class name (FQCN) of your policy. At runtime the interpreter will instantiate the policy using Class.forName(FQCN) DEFAULT cassandra.ssl.enabled Enable support for connecting to the Cassandra configured with SSL. To connect to Cassandra configured with SSL use true and provide a truststore file and password with following options. false cassandra.ssl.truststore.path Filepath for the truststore file to use for connection to Cassandra with SSL. cassandra.ssl.truststore.password Password for the truststore file to use for connection to Cassandra with SSL. Change Log3.0 (Zeppelin 0.8.2) :Update documentationUpdate interactive documentationAdd support for binary protocol V4Implement new @requestTimeOut runtime optionUpgrade Java driver version to 3.0.1Allow interpreter to add dynamic forms programmatically when using FormType.SIMPLEAllow dynamic form using default Zeppelin syntaxFixing typo on FallThroughPolicyLook for data in AngularObjectRegistry before creating dynamic formAdd missing support for ALTER statements2.0 (Zeppelin 0.8.2) :Update help menu and add changelogAdd Support for User Defined Functions, User Defined Aggregates and Materialized ViewsUpgrade Java driver version to 3.0.0-rc11.0 (Zeppelin 0.5.5-incubating) :Initial versionBugs & ContactsIf you encounter a bug for this interpreter, please create a JIRA ticket and ping me on Twitter at @doanduyhaiZeppelin Dynamic FormInterpreter Binding Mode",
"url": " /interpreter/cassandra.html",
"group": "interpreter",
"excerpt": "Apache Cassandra database is the right choice when you need scalability and high availability without compromising performance."
}
,
"/interpreter/lens.html": {
"title": "Lens Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Lens Interpreter for Apache ZeppelinOverviewApache Lens provides an Unified Analytics interface. Lens aims to cut the Data Analytics silos by providing a single view of data across multiple tiered data stores and optimal execution environment for the analytical query. It seamlessly integrates Hadoop with traditional data warehouses to appear like one.Installing and Running LensIn order to use Lens interpreters, you may install Apache Lens in some simple steps:Download Lens for latest version from the ASF. Or the older release can be found in the Archives.Before running Lens, you have to set HIVEHOME and HADOOPHOME. If you want to get more information about this, please refer to here. Lens also provides Pseudo Distributed mode. Lens pseudo-distributed setup is done by using docker. Hive server and hadoop daemons are run as separate processes in lens pseudo-distributed setup.Now, you can start lens server (or stop)../bin/lens-ctl start # (or stop)Configuring Lens InterpreterAt the "Interpreters" menu, you can edit Lens interpreter or create new one. Zeppelin provides these properties for Lens. Property Name value Description lens.client.dbname default The database schema name lens.query.enable.persistent.resultset false Whether to enable persistent resultset for queries. When enabled, server will fetch results from driver, custom format them if any and store in a configured location. The file name of query output is queryhandle-id, with configured extensions lens.server.base.url http://hostname:port/lensapi The base url for the lens server. you have to edit "hostname" and "port" that you may use(ex. http://0.0.0.0:9999/lensapi) lens.session.cluster.user default Hadoop cluster username zeppelin.lens.maxResult 1000 Max number of rows to display zeppelin.lens.maxThreads 10 If concurrency is true then how many threads? zeppelin.lens.run.concurrent true Run concurrent Lens Sessions xxx yyy anything else from [Configuring lens server](https://lens.apache.org/admin/config-server.html) Interpreter Binding for Zeppelin NotebookAfter configuring Lens interpreter, create your own notebook, then you can bind interpreters like below image.For more interpreter binding information see here.How to useYou can analyze your data by using OLAP Cube QL which is a high level SQL like language to query and describe data sets organized in data cubes.You may experience OLAP Cube like this Video tutorial.As you can see in this video, they are using Lens Client Shell(./bin/lens-cli.sh). All of these functions also can be used on Zeppelin by using Lens interpreter. Create and Use (Switch) Databases.create database newDbuse newDb Create Storage.create storage your/path/to/lens/client/examples/resources/db-storage.xml Create Dimensions, Show fields and join-chains of them.create dimension your/path/to/lens/client/examples/resources/customer.xmldimension show fields customerdimension show joinchains customer Create Caches, Show fields and join-chains of them.create cube your/path/to/lens/client/examples/resources/sales-cube.xmlcube show fields salescube show joinchains sales Create Dimtables and Fact.create dimtable your/path/to/lens/client/examples/resources/customer_table.xmlcreate fact your/path/to/lens/client/examples/resources/sales-raw-fact.xml Add partitions to Dimtable and Fact.dimtable add single-partition --dimtable_name customer_table --storage_name local --path your/path/to/lens/client/examples/resources/customer-local-part.xmlfact add partitions --fact_name sales_raw_fact --storage_name local --path your/path/to/lens/client/examples/resources/sales-raw-local-parts.xml Now, you can run queries on cubes.query execute cube select customer_city_name, product_details.description, product_details.category, product_details.color, store_sales from sales where time_range_in(delivery_time, '2015-04-11-00', '2015-04-13-00')These are just examples that provided in advance by Lens. If you want to explore whole tutorials of Lens, see the tutorial video.Lens UI ServiceLens also provides web UI service. Once the server starts up, you can open the service on http://serverhost:19999/index.html and browse. You may also check the structure that you made and use query easily here.",
"url": " /interpreter/lens.html",
"group": "interpreter",
"excerpt": "Apache Lens provides an Unified Analytics interface. Lens aims to cut the Data Analytics silos by providing a single view of data across multiple tiered data stores and optimal execution environment for the analytical query. It seamlessly integrates Hadoop with..."
}
,
"/interpreter/elasticsearch.html": {
"title": "Elasticsearch Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Elasticsearch Interpreter for Apache ZeppelinOverviewElasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements.Configuration Property Default Description elasticsearch.cluster.name elasticsearch Cluster name elasticsearch.host localhost Host of a node in the cluster elasticsearch.port 9300 Connection port ( Important: it depends on the client type, transport or http) elasticsearch.client.type transport The type of client for Elasticsearch (transport or http)( Important: the port depends on this value) elasticsearch.basicauth.username Username for a basic authentication (http) elasticsearch.basicauth.password Password for a basic authentication (http) elasticsearch.result.size 10 The size of the result set of a search query Note #1 : You can add more properties to configure the Elasticsearch client.Note #2 : If you use Shield, you can add a property named shield.user with a value containing the name and the password ( format: username:password ). For more details about Shield configuration, consult the Shield reference guide. Do not forget, to copy the shield client jar in the interpreter directory (ZEPPELIN_HOME/interpreters/elasticsearch).Enabling the Elasticsearch InterpreterIn a notebook, to enable the Elasticsearch interpreter, click the Gear icon and select Elasticsearch.Using the Elasticsearch InterpreterIn a paragraph, use %elasticsearch to select the Elasticsearch interpreter and then input all commands. To get the list of available commands, use help.%elasticsearchhelpElasticsearch interpreter:General format: <command> /<indices>/<types>/<id> <option> <JSON> - indices: list of indices separated by commas (depends on the command) - types: list of document types separated by commas (depends on the command)Commands: - search /indices/types <query> . indices and types can be omitted (at least, you have to provide '/') . a query is either a JSON-formatted query, nor a lucene query - size <value> . defines the size of the result set (default value is in the config) . if used, this command must be declared before a search command - count /indices/types <query> . same comments as for the search - get /index/type/id - delete /index/type/id - index /index/type/id <json-formatted document> . the id can be omitted, elasticsearch will generate oneTip : Use ( Ctrl + . ) for autocompletion.GetWith the get command, you can find a document by id. The result is a JSON document.%elasticsearchget /index/type/idExample:SearchWith the search command, you can send a search query to Elasticsearch. There are two formats of query:You can provide a JSON-formatted query, that is exactly what you provide when you use the REST API of Elasticsearch.See Elasticsearch search API reference document for more details about the content of the search queries.You can also provide the content of a query_string.This is a shortcut to a query like that: { "query": { "query_string": { "query": "__HERE YOUR QUERY__", "analyze_wildcard": true } } }See Elasticsearch query string syntax for more details about the content of such a query.%elasticsearchsearch /index1,index2,.../type1,type2,... <JSON document containing the query or query_string elements>If you want to modify the size of the result set, you can add a line that is setting the size, before your search command.%elasticsearchsize 50search /index1,index2,.../type1,type2,... <JSON document containing the query or query_string elements>A search query can also contain aggregations. If there is at least one aggregation, the result of the first aggregation is shown, otherwise, you get the search hits.Examples:With a JSON query:%elasticsearchsearch / { "query": { "match_all": { } } }%elasticsearchsearch /logs { "query": { "query_string": { "query": "request.method:GET AND status:200" } } }%elasticsearchsearch /logs { "aggs": {"content_length_stats": { "extended_stats": { "field": "content_length" }}} }With query_string elements:%elasticsearchsearch /logs request.method:GET AND status:200%elasticsearchsearch /logs (404 AND (POST OR DELETE))Important : a document in Elasticsearch is a JSON document, so it is hierarchical, not flat as a row in a SQL table.For the Elastic interpreter, the result of a search query is flattened.Suppose we have a JSON document:{ "date": "2015-12-08T21:03:13.588Z", "request": { "method": "GET", "url": "/zeppelin/4cd001cd-c517-4fa9-b8e5-a06b8f4056c4", "headers": [ "Accept: *.*", "Host: apache.org"] }, "status": "403", "content_length": 1234}The data will be flattened like this:content_lengthdaterequest.headers[0]request.headers[1]request.methodrequest.urlstatus12342015-12-08T21:03:13.588ZAccept: *.*Host: apache.orgGET/zeppelin/4cd001cd-c517-4fa9-b8e5-a06b8f4056c4403Examples:With a table containing the results:You can also use a predefined diagram:With a JSON query:With a JSON query containing a fields parameter (for filtering the fields in the response): in this case, all the fields values in the response are arrays, so, after flattening the result, the format of all the field names is field_name[x]With a query string:With a query containing a multi-value metric aggregation:With a query containing a multi-bucket aggregation:CountWith the count command, you can count documents available in some indices and types. You can also provide a query.%elasticsearchcount /index1,index2,.../type1,type2,... <JSON document containing the query OR a query string>Examples:Without query:With a query:IndexWith the index command, you can insert/update a document in Elasticsearch.%elasticsearchindex /index/type/id <JSON document>%elasticsearchindex /index/type <JSON document>DeleteWith the delete command, you can delete a document.%elasticsearchdelete /index/type/idApply Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form inside your queries. You can use both the text input and select form parameterization features.%elasticsearchsize ${limit=10}search /index/type { "query": { "match_all": { } } }",
"url": " /interpreter/elasticsearch.html",
"group": "interpreter",
"excerpt": "Elasticsearch is a highly scalable open-source full-text search and analytics engine."
}
,
"/interpreter/jdbc.html": {
"title": "Generic JDBC Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Generic JDBC Interpreter for Apache ZeppelinOverviewJDBC interpreter lets you create a JDBC connection to any data sources seamlessly.Inserts, Updates, and Upserts are applied immediately after running each statement.By now, it has been tested with: Postgresql - JDBC Driver Mysql - JDBC Driver MariaDB - JDBC Driver Redshift - JDBC Driver Apache Hive - JDBC Driver Apache Phoenix itself is a JDBC driver Apache Drill - JDBC Driver Apache Tajo - JDBC Driver If you are using other databases not in the above list, please feel free to share your use case. It would be helpful to improve the functionality of JDBC interpreter.Create a new JDBC InterpreterFirst, click + Create button at the top-right corner in the interpreter setting page.Fill Interpreter name field with whatever you want to use as the alias(e.g. mysql, mysql2, hive, redshift, and etc..). Please note that this alias will be used as %interpreter_name to call the interpreter in the paragraph. Then select jdbc as an Interpreter group. The default driver of JDBC interpreter is set as PostgreSQL. It means Zeppelin includes PostgreSQL driver jar in itself.So you don't need to add any dependencies(e.g. the artifact name or path for PostgreSQL driver jar) for PostgreSQL connection.The JDBC interpreter properties are defined by default like below. Name Default Value Description common.max_count 1000 The maximun number of SQL result to display default.driver org.postgresql.Driver JDBC Driver Name default.password The JDBC user password default.url jdbc:postgresql://localhost:5432/ The URL for JDBC default.user gpadmin The JDBC user name default.precode Some SQL which executes every time after initialization of the interpreter (see Binding mode) default.statementPrecode SQL code which executed before the SQL from paragraph, in the same database session (database connection) default.completer.schemaFilters Сomma separated schema (schema = catalog = database) filters to get metadata for completions. Supports '%' symbol is equivalent to any set of characters. (ex. prod_v_%,public%,info) default.completer.ttlInSeconds 120 Time to live sql completer in seconds (-1 to update everytime, 0 to disable update) default.splitQueries false Each query is executed apart and returns the result If you want to connect other databases such as Mysql, Redshift and Hive, you need to edit the property values.You can also use Credential for JDBC authentication.If default.user and default.password properties are deleted(using X button) for database connection in the interpreter setting page,the JDBC interpreter will get the account information from Credential.The below example is for Mysql connection.The last step is Dependency Setting. Since Zeppelin only includes PostgreSQL driver jar by default, you need to add each driver's maven coordinates or JDBC driver's jar file path for the other databases.That's it. You can find more JDBC connection setting examples(Mysql, MariaDB, Redshift, Apache Hive, Apache Phoenix, and Apache Tajo) in this section.More propertiesThere are more JDBC interpreter properties you can specify like below. Property Name Description common.max_result Max number of SQL result to display to prevent the browser overload. This is common properties for all connections zeppelin.jdbc.auth.type Types of authentications' methods supported are SIMPLE, and KERBEROS zeppelin.jdbc.principal The principal name to load from the keytab zeppelin.jdbc.keytab.location The path to the keytab file zeppelin.jdbc.auth.kerberos.proxy.enable When auth type is Kerberos, enable/disable Kerberos proxy with the login user to get the connection. Default value is true. default.jceks.file jceks store path (e.g: jceks://file/tmp/zeppelin.jceks) default.jceks.credentialKey jceks credential key zeppelin.jdbc.interpolation Enables ZeppelinContext variable interpolation into paragraph text. Default value is false. zeppelin.jdbc.maxConnLifetime Maximum of connection lifetime in milliseconds. A value of zero or less means the connection has an infinite lifetime. You can also add more properties by using this method.For example, if a connection needs a schema parameter, it would have to add the property as follows: name value default.schema schema_name Binding JDBC interpter to notebookTo bind the interpreters created in the interpreter setting page, click the gear icon at the top-right corner.Select(blue) or deselect(white) the interpreter buttons depending on your use cases.If you need to use more than one interpreter in the notebook, activate several buttons.Don't forget to click Save button, or you will face Interpreter *** is not found error.How to useRun the paragraph with JDBC interpreterTo test whether your databases and Zeppelin are successfully connected or not, type %jdbc_interpreter_name(e.g. %mysql) at the top of the paragraph and run show databases.%jdbc_interpreter_nameshow databasesIf the paragraph is FINISHED without any errors, a new paragraph will be automatically added after the previous one with %jdbc_interpreter_name.So you don't need to type this prefix in every paragraphs' header.Apply Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form inside your queries. You can use both the text input and select form parametrization features.%jdbc_interpreter_nameSELECT name, country, performerFROM demo.performersWHERE name='${performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}'Usage precodeYou can set precode for each data source. Code runs once while opening the connection.PropertiesAn example settings of interpreter for the two data sources, each of which has its precode parameter. Property Name Value default.driver org.postgresql.Driver default.password 1 default.url jdbc:postgresql://localhost:5432/ default.user postgres default.precode set search_path='test_path' mysql.driver com.mysql.jdbc.Driver mysql.password 1 mysql.url jdbc:mysql://localhost:3306/ mysql.user root mysql.precode set @v=12 UsageTest of execution precode for each data source.%jdbcshow search_pathReturns value of search_path which is set in the default.precode.%mysqlselect @vReturns value of v which is set in the mysql.precode.ExamplesHere are some examples you can refer to. Including the below connectors, you can connect every databases as long as it can be configured with it's JDBC driver.PostgresProperties Name Value default.driver org.postgresql.Driver default.url jdbc:postgresql://localhost:5432/ default.user mysql_user default.password mysql_password Postgres JDBC Driver DocsDependencies Artifact Excludes org.postgresql:postgresql:9.4.1211 Maven Repository: org.postgresql:postgresqlMysqlProperties Name Value default.driver com.mysql.jdbc.Driver default.url jdbc:mysql://localhost:3306/ default.user mysql_user default.password mysql_password Mysql JDBC Driver DocsDependencies Artifact Excludes mysql:mysql-connector-java:5.1.38 Maven Repository: mysql:mysql-connector-javaMariaDBProperties Name Value default.driver org.mariadb.jdbc.Driver default.url jdbc:mariadb://localhost:3306 default.user mariadb_user default.password mariadb_password MariaDB JDBC Driver DocsDependencies Artifact Excludes org.mariadb.jdbc:mariadb-java-client:1.5.4 Maven Repository: org.mariadb.jdbc:mariadb-java-clientRedshiftProperties Name Value default.driver com.amazon.redshift.jdbc42.Driver default.url jdbc:redshift://your-redshift-instance-address.redshift.amazonaws.com:5439/your-database default.user redshift_user default.password redshift_password AWS Redshift JDBC Driver DocsDependencies Artifact Excludes com.amazonaws:aws-java-sdk-redshift:1.11.51 Maven Repository: com.amazonaws:aws-java-sdk-redshiftApache HiveProperties Name Value default.driver org.apache.hive.jdbc.HiveDriver default.url jdbc:hive2://localhost:10000 default.user hive_user default.password hive_password default.proxy.user.property Example value: hive.server2.proxy.user Apache Hive 1 JDBC Driver DocsApache Hive 2 JDBC Driver DocsDependencies Artifact Excludes org.apache.hive:hive-jdbc:0.14.0 org.apache.hadoop:hadoop-common:2.6.0 Maven Repository : org.apache.hive:hive-jdbcImpersonationWhen Zeppelin server is running with authentication enabled, then the interpreter can utilize Hive's user proxy feature i.e. send extra parameter for creating and running a session ("hive.server2.proxy.user=": "${loggedInUser}"). This is particularly useful when multiple users are sharing a notebook.To enable this set following:zeppelin.jdbc.auth.type as SIMPLE or KERBEROS (if required) in the interpreter setting.${prefix}.proxy.user.property as hive.server2.proxy.userSee User Impersonation in interpreter for more information.Sample configuration Name Value hive.driver org.apache.hive.jdbc.HiveDriver hive.password hive.url jdbc:hive2://hive-server-host:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 hive.proxy.user.property hive.server2.proxy.user zeppelin.jdbc.auth.type SIMPLE Apache PhoenixPhoenix supports thick and thin connection types:Thick client is faster, but must connect directly to ZooKeeper and HBase RegionServers.Thin client has fewer dependencies and connects through a Phoenix Query Server instance.Use the appropriate default.driver, default.url, and the dependency artifact for your connection type.Thick client connectionProperties Name Value default.driver org.apache.phoenix.jdbc.PhoenixDriver default.url jdbc:phoenix:localhost:2181:/hbase-unsecure default.user phoenix_user default.password phoenix_password Dependencies Artifact Excludes org.apache.phoenix:phoenix-core:4.4.0-HBase-1.0 Maven Repository: org.apache.phoenix:phoenix-coreThin client connectionProperties Name Value default.driver org.apache.phoenix.queryserver.client.Driver default.url jdbc:phoenix:thin:url=http://localhost:8765;serialization=PROTOBUF default.user phoenix_user default.password phoenix_password DependenciesBefore Adding one of the below dependencies, check the Phoenix version first. Artifact Excludes Description org.apache.phoenix:phoenix-server-client:4.7.0-HBase-1.1 For Phoenix 4.7 org.apache.phoenix:phoenix-queryserver-client:4.8.0-HBase-1.2 For Phoenix 4.8+ Maven Repository: org.apache.phoenix:phoenix-queryserver-clientApache TajoProperties Name Value default.driver org.apache.tajo.jdbc.TajoDriver default.url jdbc:tajo://localhost:26002/default Apache Tajo JDBC Driver DocsDependencies Artifact Excludes org.apache.tajo:tajo-jdbc:0.11.0 Maven Repository: org.apache.tajo:tajo-jdbcObject InterpolationThe JDBC interpreter also supports interpolation of ZeppelinContext objects into the paragraph text.The following example shows one use of this facility:In Scala cell:z.put("country_code", "KR") // ...In later JDBC cell:%jdbc_interpreter_nameselect * from patents_list where priority_country = '{country_code}' and filing_date like '2015-%'Object interpolation is disabled by default, and can be enabled for all instances of the JDBC interpreter by setting the value of the property zeppelin.jdbc.interpolation to true (see More Properties above). More details of this feature can be found in the Spark interpreter documentation under Zeppelin-ContextBug reportingIf you find a bug using JDBC interpreter, please create a JIRA ticket.",
"url": " /interpreter/jdbc.html",
"group": "interpreter",
"excerpt": "Generic JDBC Interpreter lets you create a JDBC connection to any data source. You can use Postgres, MySql, MariaDB, Redshift, Apache Hive, Apache Phoenix, Apache Drill and Apache Tajo using JDBC interpreter."
}
,
"/interpreter/neo4j.html": {
"title": "Neo4j Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Neo4j Interpreter for Apache ZeppelinOverviewNeo4j is a native graph database, designed to store and process graphs from bottom to top.Configuration Property Default Description neo4j.url bolt://localhost:7687 The Neo4j's BOLT url. neo4j.auth.type BASIC The Neo4j's authentication type (NONE, BASIC). neo4j.auth.user neo4j The Neo4j user name. neo4j.auth.password neo4j The Neo4j user password. neo4j.max.concurrency 50 Max concurrency call from Zeppelin to Neo4j server. Enabling the Neo4j InterpreterIn a notebook, to enable the Neo4j interpreter, click the Gear icon and select Neo4j.Using the Neo4j InterpreterIn a paragraph, use %neo4j to select the Neo4j interpreter and then input the Cypher commands.For list of Cypher commands please refer to the official Cyper Refcard%neo4j//Sample the TrumpWorld datasetWITH'https://docs.google.com/spreadsheets/u/1/d/1Z5Vo5pbvxKJ5XpfALZXvCzW26Cl4we3OaN73K9Ae5Ss/export?format=csv&gid=1996904412' AS urlLOAD CSV WITH HEADERS FROM url AS rowRETURN row.`Entity A`, row.`Entity A Type`, row.`Entity B`, row.`Entity B Type`, row.Connection, row.`Source(s)`LIMIT 10The Neo4j interpreter leverages the Network display system allowing to visualize the them directly from the paragraph.Write your Cypher queries and navigate your graphThis query:%neo4jMATCH (vp:Person {name:"VLADIMIR PUTIN"}), (dt:Person {name:"DONALD J. TRUMP"})MATCH path = allShortestPaths( (vp)-[*]-(dt) )RETURN pathproduces the following result_Apply Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form inside your queries. This query:%neo4jMATCH (o:Organization)-[r]-()RETURN o.name, count(*), collect(distinct type(r)) AS typesORDER BY count(*) DESCLIMIT ${Show top=10}produces the following result:",
"url": " /interpreter/neo4j.html",
"group": "interpreter",
"excerpt": "Neo4j is a native graph database, designed to store and process graphs from bottom to top."
}
,
"/interpreter/bigquery.html": {
"title": "BigQuery Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->BigQuery Interpreter for Apache ZeppelinOverviewBigQuery is a highly scalable no-ops data warehouse in the Google Cloud Platform. Querying massive datasets can be time consuming and expensive without the right hardware and infrastructure. Google BigQuery solves this problem by enabling super-fast SQL queries against append-only tables using the processing power of Google's infrastructure. Simply move your data into BigQuery and let us handle the hard work. You can control access to both the project and your data based on your business needs, such as giving others the ability to view or query your data. Configuration Name Default Value Description zeppelin.bigquery.project_id Google Project Id zeppelin.bigquery.wait_time 5000 Query Timeout in Milliseconds zeppelin.bigquery.max_no_of_rows 100000 Max result set size zeppelin.bigquery.sql_dialect BigQuery SQL dialect (standardSQL or legacySQL). If empty, [query prefix](https://cloud.google.com/bigquery/docs/reference/standard-sql/enabling-standard-sql#sql-prefix) like '#standardSQL' can be used. BigQuery APIZeppelin is built against BigQuery API version v2-rev265-1.21.0 - API JavadocsEnabling the BigQuery InterpreterIn a notebook, to enable the BigQuery interpreter, click the Gear icon and select bigquery.Provide Application Default CredentialsWithin Google Cloud Platform (e.g. Google App Engine, Google Compute Engine),built-in credentials are used by default.Outside of GCP, follow the Google API authentication instructions for Zeppelin Google Cloud StorageUsing the BigQuery InterpreterIn a paragraph, use %bigquery.sql to select the BigQuery interpreter and then input SQL statements against your datasets stored in BigQuery.You can use BigQuery SQL Reference to build your own SQL.For Example, SQL to query for top 10 departure delays across airports using the flights public dataset%bigquery.sqlSELECT departure_airport,count(case when departure_delay>0 then 1 else 0 end) as no_of_delays FROM [bigquery-samples:airline_ontime_data.flights] group by departure_airport order by 2 desc limit 10Another Example, SQL to query for most commonly used java packages from the github data hosted in BigQuery %bigquery.sqlSELECT package, COUNT(*) countFROM ( SELECT REGEXP_EXTRACT(line, r' ([a-z0-9._]*).') package, id FROM ( SELECT SPLIT(content, 'n') line, id FROM [bigquery-public-data:github_repos.sample_contents] WHERE content CONTAINS 'import' AND sample_path LIKE '%.java' HAVING LEFT(line, 6)='import' ) GROUP BY package, id )GROUP BY 1ORDER BY count DESCLIMIT 40Technical descriptionFor in-depth technical details on current implementation please refer to bigquery/README.md.",
"url": " /interpreter/bigquery.html",
"group": "interpreter",
"excerpt": "BigQuery is a highly scalable no-ops data warehouse in the Google Cloud Platform."
}
,
"/interpreter/scio.html": {
"title": "Scio Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Scio Interpreter for Apache ZeppelinOverviewScio is a Scala DSL for Google Cloud Dataflow and Apache Beam inspired by Spark and Scalding. See the current wiki and API documentation for more information.Configuration Name Default Value Description zeppelin.scio.argz --runner=InProcessPipelineRunner Scio interpreter wide arguments. Documentation: https://github.com/spotify/scio/wiki#options and https://cloud.google.com/dataflow/pipelines/specifying-exec-params zeppelin.scio.maxResult 1000 Max number of SCollection results to display Enabling the Scio InterpreterIn a notebook, to enable the Scio interpreter, click the Gear icon and select beam (beam.scio).Using the Scio InterpreterIn a paragraph, use %beam.scio to select the Scio interpreter. You can use it much the same way as vanilla Scala REPL and Scio REPL. State (like variables, imports, execution etc) is shared among all Scio paragraphs. There is a special variable argz which holds arguments from Scio interpreter settings. The easiest way to proceed is to create a Scio context via standard ContextAndArgs.%beam.scioval (sc, args) = ContextAndArgs(argz)Use sc context the way you would in a regular pipeline/REPL.Example:%beam.scioval (sc, args) = ContextAndArgs(argz)sc.parallelize(Seq("foo", "foo", "bar")).countByValue.closeAndDisplay()If you close Scio context, go ahead an create a new one using ContextAndArgs. Please refer to Scio wiki for more complex examples. You can close Scio context much the same way as in Scio REPL, and use Zeppelin display helpers to synchronously close and display results - read more below.ProgressThere can be only one paragraph running at once. There is no notion of overall progress, thus progress bar will show 0.SCollection display helpersScio interpreter comes with display helpers to ease working with Zeppelin notebooks. Simply use closeAndDisplay() on SCollection to close context and display the results. The number of results is limited by zeppelin.scio.maxResult (by default 1000).Supported SCollection types:Scio's typed BigQueryScala's Products (case classes, tuples)Google BigQuery's TableRowApache AvroAll Scala's AnyValHelper methodsThere are different helper methods for different objects. You can easily display results from SCollection, Future[Tap] and Tap.SCollection helperSCollection has closeAndDisplay Zeppelin helper method for types listed above. Use it to synchronously close Scio context, and once available pull and display results.Future[Tap] helperFuture[Tap] has waitAndDisplay Zeppelin helper method for types listed above. Use it to synchronously wait for results, and once available pull and display results.Tap helperTap has display Zeppelin helper method for types listed above. Use it to pull and display results.ExamplesBigQuery example:%beam.scio@BigQueryType.fromQuery("""|SELECT departure_airport,count(case when departure_delay>0 then 1 else 0 end) as no_of_delays |FROM [bigquery-samples:airline_ontime_data.flights] |group by departure_airport |order by 2 desc |limit 10""".stripMargin) class Flightsval (sc, args) = ContextAndArgs(argz)sc.bigQuerySelect(Flights.query).closeAndDisplay(Flights.schema)BigQuery typed example:%beam.scio@BigQueryType.fromQuery("""|SELECT departure_airport,count(case when departure_delay>0 then 1 else 0 end) as no_of_delays |FROM [bigquery-samples:airline_ontime_data.flights] |group by departure_airport |order by 2 desc |limit 10""".stripMargin) class Flightsval (sc, args) = ContextAndArgs(argz)sc.typedBigQuery[Flights]().flatMap(_.no_of_delays).mean.closeAndDisplay()Avro example:%beam.scioimport com.spotify.data.ExampleAvroval (sc, args) = ContextAndArgs(argz)sc.avroFile[ExampleAvro]("gs://<bucket>/tmp/my.avro").take(10).closeAndDisplay()Avro example with a view schema:%beam.scioimport com.spotify.data.ExampleAvroimport org.apache.avro.Schemaval (sc, args) = ContextAndArgs(argz)val view = Schema.parse("""{"type":"record","name":"ExampleAvro","namespace":"com.spotify.data","fields":[{"name":"track","type":"string"}, {"name":"artist", "type":"string"}]}""")sc.avroFile[EndSongCleaned]("gs://<bucket>/tmp/my.avro").take(10).closeAndDisplay(view)Google credentialsScio Interpreter will try to infer your Google Cloud credentials from its environment, it will take into the account:argz interpreter settings (doc)environment variable (GOOGLE_APPLICATION_CREDENTIALS)gcloud configurationBigQuery macro credentialsCurrently BigQuery project for macro expansion is inferred using Google Dataflow's DefaultProjectFactory().create()",
"url": " /interpreter/scio.html",
"group": "interpreter",
"excerpt": "Scio is a Scala DSL for Apache Beam/Google Dataflow model."
}
,
"/interpreter/shell.html": {
"title": "Shell interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Shell interpreter for Apache ZeppelinOverviewShell interpreter uses Apache Commons Exec to execute external processes. In Zeppelin notebook, you can use %sh in the beginning of a paragraph to invoke system shell and run commands.Note : Currently each command runs as the user Zeppelin server is running as.ConfigurationAt the "Interpreters" menu in Zeppelin dropdown menu, you can set the property value for Shell interpreter. Name Default Description shell.command.timeout.millisecs 60000 Shell command time out in millisecs shell.working.directory.user.home false If this set to true, the shell's working directory will be set to user home zeppelin.shell.auth.type Types of authentications' methods supported are SIMPLE, and KERBEROS zeppelin.shell.principal The principal name to load from the keytab zeppelin.shell.keytab.location The path to the keytab file zeppelin.shell.interpolation false Enable ZeppelinContext variable interpolation into paragraph text ExampleThe following example demonstrates the basic usage of Shell in a Zeppelin notebook.If you need further information about Zeppelin Interpreter Setting for using Shell interpreter, please read What is interpreter setting? section first.Kerberos refresh intervalFor changing the default behavior of when to renew Kerberos ticket following changes can be made in conf/zeppelin-env.sh.# Change Kerberos refresh interval (default value is 1d). Allowed postfix are ms, s, m, min, h, and d.export LAUNCH_KERBEROS_REFRESH_INTERVAL=4h# Change kinit number retries (default value is 5), which means if the kinit command fails for 5 retries consecutively it will close the interpreter. export KINIT_FAIL_THRESHOLD=10Object InterpolationThe shell interpreter also supports interpolation of ZeppelinContext objects into the paragraph text.The following example shows one use of this facility:In Scala cell:z.put("dataFileName", "members-list-003.parquet") // ...val members = spark.read.parquet(z.get("dataFileName")) // ...In later Shell cell:%shrm -rf {dataFileName}Object interpolation is disabled by default, and can be enabled (for the Shell interpreter) by setting the value of the property zeppelin.shell.interpolation to true (see Configuration above). More details of this feature can be found in Zeppelin-Context",
"url": " /interpreter/shell.html",
"group": "interpreter",
"excerpt": "Shell interpreter uses Apache Commons Exec to execute external processes."
}
,
"/interpreter/scalding.html": {
"title": "Scalding Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Scalding Interpreter for Apache ZeppelinScalding is an open source Scala library for writing MapReduce jobs.Building the Scalding InterpreterYou have to first build the Scalding interpreter by enable the scalding profile as follows:mvn clean package -Pscalding -DskipTestsEnabling the Scalding InterpreterIn a notebook, to enable the Scalding interpreter, click on the Gear icon,select Scalding, and hit Save.Configuring the InterpreterScalding interpreter runs in two modes:localhdfsIn the local mode, you can access files on the local server and scalding transformation are done locally.In hdfs mode you can access files in HDFS and scalding transformation are run as hadoop map-reduce jobs.Zeppelin comes with a pre-configured Scalding interpreter in local mode.To run the scalding interpreter in the hdfs mode you have to do the following:Set the classpath with ZEPPELIN_CLASSPATH_OVERRIDESIn conf/zeppelinenv.sh, you have to setZEPPELINCLASSPATH_OVERRIDES to the contents of 'hadoop classpath'and directories with custom jar files you need for your scalding commands.Set arguments to the scalding replThe default arguments are: --local --replFor hdfs mode you need to add: --hdfs --replIf you want to add custom jars, you need to add: -libjars directory/*:directory/*For reducer estimation, you need to add something like:-Dscalding.reducer.estimator.classes=com.twitter.scalding.reducer_estimation.InputSizeReducerEstimatorSet max.open.instancesIf you want to control the maximum number of open interpreters, you have to select "scoped" interpreter for noteoption and set max.open.instances argument.Testing the InterpreterLocal modeIn example, by using the Alice in Wonderland tutorial, we will count words (of course!), and plot a graph of the top 10 words in the book.%scaldingimport scala.io.Source// Get the Alice in Wonderland book from gutenberg.org:val alice = Source.fromURL("http://www.gutenberg.org/files/11/11.txt").getLinesval aliceLineNum = alice.zipWithIndex.toListval alicePipe = TypedPipe.from(aliceLineNum)// Now get a list of words for the book:val aliceWords = alicePipe.flatMap { case (text, _) => text.split("s+").toList }// Now lets add a count for each word:val aliceWithCount = aliceWords.filterNot(_.equals("")).map { word => (word, 1L) }// let's sum them for each word:val wordCount = aliceWithCount.group.sumprint ("Here are the top 10 wordsn")val top10 = wordCount .groupAll .sortBy { case (word, count) => -count } .take(10)top10.dump%scaldingval table = "wordst countn" + top10.toIterator.map{case (k, (word, count)) => s"$wordt$count"}.mkString("n")print("%table " + table)If you click on the icon for the pie chart, you should be able to see a chart like this:HDFS modeTest mode%scaldingmodeThis command should print:res4: com.twitter.scalding.Mode = Hdfs(true,Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml)Test HDFS readval testfile = TypedPipe.from(TextLine("/user/x/testfile"))testfile.dumpThis command should print the contents of the hdfs file /user/x/testfile.Test map-reduce jobval testfile = TypedPipe.from(TextLine("/user/x/testfile"))val a = testfile.groupAll.size.valuesa.toListThis command should create a map reduce job.Future WorkBetter user feedback (hadoop url, progress updates)Ability to cancel jobsAbility to dynamically load jars without restarting the interpreterMultiuser scalability (run scalding interpreters on different servers)",
"url": " /interpreter/scalding.html",
"group": "interpreter",
"excerpt": "Scalding is an open source Scala library for writing MapReduce jobs."
}
,
"/interpreter/alluxio.html": {
"title": "Alluxio Interpreter for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Alluxio Interpreter for Apache ZeppelinOverviewAlluxio is a memory-centric distributed storage system enabling reliable data sharing at memory-speed across cluster frameworks.Configuration Name Class Description alluxio.master.hostname localhost Alluxio master hostname alluxio.master.port 19998 Alluxio master port Enabling Alluxio InterpreterIn a notebook, to enable the Alluxio interpreter, click on the Gear icon and select Alluxio.Using the Alluxio InterpreterIn a paragraph, use %alluxio to select the Alluxio interpreter and then input all commands.%alluxiohelpTip : Use ( Ctrl + . ) for autocompletion.Interpreter CommandsThe Alluxio interpreter accepts the following commands. Operation Syntax Description cat cat "path" Print the content of the file to the console. chgrp chgrp "group" "path" Change the group of the directory or file. chmod chmod "permission" "path" Change the permission of the directory or file. chown chown "owner" "path" Change the owner of the directory or file. copyFromLocal copyFromLocal "source path" "remote path" Copy the specified file specified by "source path" to the path specified by "remote path". This command will fail if "remote path" already exists. copyToLocal copyToLocal "remote path" "local path" Copy the specified file from the path specified by "remote path" to a local destination. count count "path" Display the number of folders and files matching the specified prefix in "path". du du "path" Display the size of a file or a directory specified by the input path. fileInfo fileInfo "path" Print the information of the blocks of a specified file. free free "path" Free a file or all files under a directory from Alluxio. If the file/directory is also in under storage, it will still be available there. getCapacityBytes getCapacityBytes Get the capacity of the AlluxioFS. getUsedBytes getUsedBytes Get number of bytes used in the AlluxioFS. load load "path" Load the data of a file or a directory from under storage into Alluxio. loadMetadata loadMetadata "path" Load the metadata of a file or a directory from under storage into Alluxio. location location "path" Display a list of hosts that have the file data. ls ls "path" List all the files and directories directly under the given path with information such as size. mkdir mkdir "path1" ... "pathn" Create directory(ies) under the given paths, along with any necessary parent directories. Multiple paths separated by spaces or tabs. This command will fail if any of the given paths already exist. mount mount "path" "uri" Mount the underlying file system path "uri" into the Alluxio namespace as "path". The "path" is assumed not to exist and is created by the operation. No data or metadata is loaded from under storage into Alluxio. After a path is mounted, operations on objects under the mounted path are mirror to the mounted under storage. mv mv "source" "destination" Move a file or directory specified by "source" to a new location "destination". This command will fail if "destination" already exists. persist persist "path" Persist a file or directory currently stored only in Alluxio to the underlying file system. pin pin "path" Pin the given file to avoid evicting it from memory. If the given path is a directory, it recursively pins all the files contained and any new files created within this directory. report report "path" Report to the master that a file is lost. rm rm "path" Remove a file. This command will fail if the given path is a directory rather than a file. setTtl setTtl "time" Set the TTL (time to live) in milliseconds to a file. tail tail "path" Print the last 1KB of the specified file to the console. touch touch "path" Create a 0-byte file at the specified location. unmount unmount "path" Unmount the underlying file system path mounted in the Alluxio namespace as "path". Alluxio objects under "path" are removed from Alluxio, but they still exist in the previously mounted under storage. unpin unpin "path" Unpin the given file to allow Alluxio to evict this file again. If the given path is a directory, it recursively unpins all files contained and any new files created within this directory. unsetTtl unsetTtl Remove the TTL (time to live) setting from a file. How to test it's workingBe sure to have configured correctly the Alluxio interpreter, then open a new paragraph and type one of the above commands.Below a simple example to show how to interact with Alluxio interpreter.Following steps are performed:using sh interpreter a new text file is created on local machineusing Alluxio interpreter:is listed the content of the afs (Alluxio File System) rootthe file previously created is copied to afsis listed again the content of the afs root to check the existence of the new copied fileis showed the content of the copied file (using the tail command)the file previously copied to afs is copied to local machine using sh interpreter it's checked the existence of the new file copied from Alluxio and its content is showed ",
"url": " /interpreter/alluxio.html",
"group": "interpreter",
"excerpt": "Alluxio is a memory-centric distributed storage system enabling reliable data sharing at memory-speed across cluster frameworks."
}
,
"/development/helium/writing_visualization_transformation.html": {
"title": "Transformations in Zeppelin Visualization",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Transformations for Zeppelin VisualizationOverviewTransformations renders setting which allows users to set columns and transforms table rows according to the configured columns.Zeppelin provides 4 types of transformations.1. PassthroughTransformationPassthroughTransformation is the simple transformation which does not convert original tabledata at all.See passthrough.js2. ColumnselectorTransformationColumnselectorTransformation is uses when you need N axes but do not need aggregation. See columnselector.js3. PivotTransformationPivotTransformation provides group by and aggregation. Every chart using PivotTransformation has 3 axes. Keys, Groups and Values.See pivot.js4. AdvancedTransformationAdvancedTransformation has more detailed options while providing existing features of PivotTransformation and ColumnselectorTransformationmultiple sub chartsconfigurable chart axesparameter widgets: input, checkbox, option, textareaparsing parameters automatically based on their typesexpand / fold axis and parameter panelsmultiple transformation methods while supporting lazy converting re-initialize the whole configuration based on spec hash.SpecAdvancedTransformation requires spec which includes axis and parameter details for charts.Let's create 2 sub-charts called line and no-group. Each sub chart can have different axis and parameter depending on their requirements.class AwesomeVisualization extends Visualization { constructor(targetEl, config) { super(targetEl, config) const spec = { charts: { 'line': { transform: { method: 'object', }, sharedAxis: false, /** set if you want to share axes between sub charts, default is `false` */ axis: { 'xAxis': { dimension: 'multiple', axisType: 'key', description: 'serial', }, 'yAxis': { dimension: 'multiple', axisType: 'aggregator', description: 'serial', }, 'category': { dimension: 'multiple', axisType: 'group', description: 'categorical', }, }, parameter: { 'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', }, 'yAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of yAxis', }, 'lineWidth': { valueType: 'int', defaultValue: 0, description: 'width of line', }, }, }, 'no-group': { transform: { method: 'object', }, sharedAxis: false, axis: { 'xAxis': { dimension: 'single', axisType: 'key', }, 'yAxis': { dimension: 'multiple', axisType: 'value', }, }, parameter: { 'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', }, 'yAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of yAxis', }, }, }, } this.transformation = new AdvancedTransformation(config, spec) } ... // `render` will be called whenever `axis` or `parameter` is changed render(data) { const { chart, parameter, column, transformer, } = data if (chart === 'line') { const transformed = transformer() // draw line chart } else if (chart === 'no-group') { const transformed = transformer() // draw no-group chart } }}Spec: axisField NameAvailable Values (type)DescriptiondimensionsingleAxis can contains only 1 columndimensionmultipleAxis can contains multiple columnsaxisTypekeyColumn(s) in this axis will be used as key like in PivotTransformation. These columns will be served in column.keyaxisTypeaggregatorColumn(s) in this axis will be used as value like in PivotTransformation. These columns will be served in column.aggregatoraxisTypegroupColumn(s) in this axis will be used as group like in PivotTransformation. These columns will be served in column.groupaxisType(string)Any string value can be used here. These columns will be served in column.custommaxAxisCount (optional)(int)The max number of columns that this axis can contain. (unlimited if undefined)minAxisCount (optional)(int)The min number of columns that this axis should contain to draw chart. (1 in case of single dimension)description (optional)(string)Description for the axis.Here is an example.axis: { 'xAxis': { dimension: 'multiple', axisType: 'key', }, 'yAxis': { dimension: 'multiple', axisType: 'aggregator'}, 'category': { dimension: 'multiple', axisType: 'group', maxAxisCount: 2, valueType: 'string', },},Spec: sharedAxisIf you set sharedAxis: false for sub charts, then their axes are persisted in global space (shared). It's useful for when you creating multiple sub charts sharing their axes but have different parameters. For example, basic-column, stacked-column, percent-columnpie and donutHere is an example. const spec = { charts: { 'column': { transform: { method: 'array', }, sharedAxis: true, axis: { ... }, parameter: { ... }, }, 'stacked': { transform: { method: 'array', }, sharedAxis: true, axis: { ... } parameter: { ... }, },Spec: parameterField NameAvailable Values (type)DescriptionvalueTypestringParameter which has string valuevalueTypeintParameter which has int valuevalueTypefloatParameter which has float valuevalueTypebooleanParameter which has boolean value used with checkbox widget usuallyvalueTypeJSONParameter which has JSON value used with textarea widget usually. defaultValue should be "" (empty string). Thisdescription(string)Description of this parameter. This value will be parsed as HTML for pretty outputwidgetinputUse input widget. This is the default widget (if widget is undefined)widgetcheckboxUse checkbox widget.widgettextareaUse textarea widget.widgetoptionUse select + option widget. This parameter should have optionValues field as well.optionValues(Array)Available option values used with the option widgetHere is an example.parameter: { // string type, input widget 'xAxisUnit': { valueType: 'string', defaultValue: '', description: 'unit of xAxis', }, // boolean type, checkbox widget 'inverted': { widget: 'checkbox', valueType: 'boolean', defaultValue: false, description: 'invert x and y axes', }, // string type, option widget with `optionValues` 'graphType': { widget: 'option', valueType: 'string', defaultValue: 'line', description: 'graph type', optionValues: [ 'line', 'smoothedLine', 'step', ], }, // HTML in `description` 'dateFormat': { valueType: 'string', defaultValue: '', description: 'format of date (<a href="https://docs.amcharts.com/3/javascriptcharts/AmGraph#dateFormat">doc</a>) (e.g YYYY-MM-DD)', }, // JSON type, textarea widget 'yAxisGuides': { widget: 'textarea', valueType: 'JSON', defaultValue: '', description: 'guides of yAxis ', },Spec: transformField NameAvailable Values (type)Descriptionmethodobjectdesigned for rows requiring object manipulationmethodarraydesigned for rows requiring array manipulationmethodarray:2-keydesigned for xyz charts (e.g bubble chart)methoddrill-downdesigned for drill-down chartsmethodrawwill return the original tableData.rowsWhatever you specified as transform.method, the transformer value will be always function for lazy computation. // advanced-transformation.util#getTransformerif (transformSpec.method === 'raw') { transformer = () => { return rows; }} else if (transformSpec.method === 'array') { transformer = () => { ... return { ... } }}Here is actual usage.class AwesomeVisualization extends Visualization { constructor(...) { /** setup your spec */ } ... // `render` will be called whenever `axis` or `parameter` are changed render(data) { const { chart, parameter, column, transformer, } = data if (chart === 'line') { const transformed = transformer() // draw line chart } else if (chart === 'no-group') { const transformed = transformer() // draw no-group chart } } ...}",
"url": " /development/helium/writing_visualization_transformation.html",
"group": "development/helium",
"excerpt": "Description for Transformations"
}
,
"/development/helium/writing_spell.html": {
"title": "Writing a new Helium Spell",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Writing a new SpellWhat is Apache Zeppelin SpellSpell is a kind of interpreter that runs on browser not on backend. So, technically it's the frontend interpreter.It can provide many benefits.Spell is pluggable frontend interpreter. So it can be installed and removed easily using helium registry.Every spell is written in javascript. It means you can use existing javascript libraries whatever you want.Spell runs on browser like display system (%html, %table). In other words, every spell can be used as display system as well.How it worksHelium Spell works like Helium Visualization.Every helium packages are loaded from central (online) registry or local registryYou can see loaded packages in /helium page.When you enable a spell, it's built from server and sent to clientFinally it will be loaded into browser.How to use spell1. EnablingFind a spell what you want to use in /helium package and click Enable button.2. UsingSpell works like an interpreter. Use the MAGIC value to execute spell in a note. (you might need to refresh after enabling)For example, Use %echo for the Echo Spell.Write a new SpellMaking a new spell is similar to Helium Visualization#write-new-visualization.Add framework dependency called zeppelin-spell into package.jsonWrite code using frameworkPublish your spell to npm1. Create a npm packageCreate a package.json in new directory for spell.You have to add a framework called zeppelin-spell as a dependency to create spell (zeppelin-spell)Also, you can add any dependencies you want to utilise.Here's an example{ "name": "zeppelin-echo-spell", "description": "Zeppelin Echo Spell (example)", "version": "1.0.0", "main": "index", "author": "", "license": "Apache-2.0", "dependencies": { "zeppelin-spell": "*" }, "helium": { "icon" : "<i class='fa fa-repeat'></i>", "spell": { "magic": "%echo", "usage": "%echo <TEXT>" } }}2. Write spell using frameworkHere are some examples you can referEcho SpellMarkdown Spell: Using libraryFlowchart Spell: Using DOMGoogle Translation API Spell: Using API (returning promise)Now, you need to write code to create spell which processing text.import { SpellBase, SpellResult, DefaultDisplayType,} from 'zeppelin-spell';export default class EchoSpell extends SpellBase { constructor() { /** pass magic to super class's constructor parameter */ super("%echo"); } interpret(paragraphText) { const processed = paragraphText + '!'; /** * should return `SpellResult` which including `data` and `type` * default type is `TEXT` if you don't specify. */ return new SpellResult(processed); }}Here is another example. Let's say we want to create markdown spell. First of all, we should add a dependency for markdown in package.json// package.json "dependencies": { "markdown": "0.5.0", "zeppelin-spell": "*" },And here is spell code.import { SpellBase, SpellResult, DefaultDisplayType,} from 'zeppelin-spell';import md from 'markdown';const markdown = md.markdown;export default class MarkdownSpell extends SpellBase { constructor() { super("%markdown"); } interpret(paragraphText) { const parsed = markdown.toHTML(paragraphText); /** * specify `DefaultDisplayType.HTML` since `parsed` will contain DOM * otherwise it will be rendered as `DefaultDisplayType.TEXT` (default) */ return new SpellResult(parsed, DefaultDisplayType.HTML); }}You might want to manipulate DOM directly (e.g google d3.js), then refer Flowchart SpellYou might want to return promise not string (e.g API call), then refer Google Translation API Spell3. Create Helium package file for local deploymentYou don't want to publish your package every time you make a change in your spell. Zeppelin provides local deploy.The only thing you need to do is creating a Helium Package file (JSON) for local deploy.It's automatically created when you publish to npm repository but in local case, you should make it by yourself.{ "type" : "SPELL", "name" : "zeppelin-echo-spell", "version": "1.0.0", "description" : "Return just what receive (example)", "artifact" : "./zeppelin-examples/zeppelin-example-spell-echo", "license" : "Apache-2.0", "spell": { "magic": "%echo", "usage": "%echo <TEXT>" }}Place this file in your local registry directory (default $ZEPPELIN_HOME/helium).type should be SPELLMake sure that artifact should be same as your spell directory.You can get information about other fields in Helium Visualization#3-create-helium-package-file-and-locally-deploy.4. Run in dev modecd zeppelin-webyarn run dev:heliumYou can browse localhost:9000. Every time refresh your browser, Zeppelin will rebuild your spell and reload changes.5. Publish your spell to the npm repositorySee Publishing npm packages",
"url": " /development/helium/writing_spell.html",
"group": "development/helium",
"excerpt": "Spell is a kind of interpreter that runs on browser not on backend. So, technically it's the frontend interpreter. "
}
,
"/development/helium/writing_visualization_basic.html": {
"title": "Writing a new Helium Visualization: basic",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Writing a new VisualizationWhat is Apache Zeppelin VisualizationApache Zeppelin Visualization is a pluggable package that can be loaded/unloaded on runtime through Helium framework in Zeppelin. A Visualization is a javascript npm package and user can use them just like any other built-in visualization in notebook.How it works1. Load Helium package files from registryZeppelin needs to know what Visualization packages are available. Zeppelin will read information of packages from both online and local registry.Registries are configurable through ZEPPELIN_HELIUM_LOCALREGISTRY_DEFAULT env variable or zeppelin.helium.localregistry.default property.2. Enable packagesOnce Zeppelin loads Helium package files from registries, available packages are displayed in Helium menu.Click 'enable' button.3. Create and load visualization bundle on the flyOnce a Visualization package is enabled, HeliumBundleFactory creates a js bundle. The js bundle is served by helium/bundle/load rest api endpoint.4. Run visualizationZeppelin shows additional button for loaded Visualizations.User can use just like any other built-in visualizations.Write new Visualization1. Create a npm packageCreate a package.json in your new Visualization directory. You can add any dependencies in package.json, but you must include two dependencies: zeppelin-vis and zeppelin-tabledata.Here's an example{ "name": "zeppelin_horizontalbar", "description" : "Horizontal Bar chart", "version": "1.0.0", "main": "horizontalbar", "author": "", "license": "Apache-2.0", "dependencies": { "zeppelin-tabledata": "*", "zeppelin-vis": "*" }}2. Create your own visualizationTo create your own visualization, you need to create a js file and import Visualization class from zeppelin-vis package and extend the class. zeppelin-tabledata package provides some useful transformations, like pivot, you can use in your visualization. (you can create your own transformation, too).Visualization class, there're several methods that you need to override and implement. Here's simple visualization that just prints Hello world.import Visualization from 'zeppelin-vis'import PassthroughTransformation from 'zeppelin-tabledata/passthrough'export default class helloworld extends Visualization { constructor(targetEl, config) { super(targetEl, config) this.passthrough = new PassthroughTransformation(config); } render(tableData) { this.targetEl.html('Hello world!') } getTransformation() { return this.passthrough }}To learn more about Visualization class, check visualization.js.You can check complete visualization package example here.Zeppelin's built-in visualization uses the same API, so you can check built-in visualizations as additional examples.3. Create Helium package file and locally deployHelium Package file is a json file that provides information about the application.Json file contains the following information{ "type" : "VISUALIZATION", "name" : "zeppelin_horizontalbar", "description" : "Horizontal Bar chart (example)", "license" : "Apache-2.0", "artifact" : "./zeppelin-examples/zeppelin-example-horizontalbar", "icon" : "<i class='fa fa-bar-chart rotate90flipX'></i>"}Place this file in your local registry directory (default ./helium).typeWhen you're creating a visualization, 'type' should be 'VISUALIZATION'. Check these types as well.Helium ApplicationHelium SpellnameName of visualization. Should be unique. Allows [A-Za-z90-9_].descriptionA short description about visualization.artifactLocation of the visualization npm package. Support npm package with version or local filesystem path.e.g.When artifact exists in npm repository"artifact": "my-visualiztion@1.0.0"When artifact exists in local file system"artifact": "/path/to/my/visualization"licenseLicense information.e.g."license": "Apache-2.0"iconIcon to be used in visualization select button. String in this field will be rendered as a HTML tag.e.g."icon": "<i class='fa fa-coffee'></i>"4. Run in dev modePlace your Helium package file in local registry (ZEPPELIN_HOME/helium).Run Zeppelin. And then run zeppelin-web in visualization dev mode.cd zeppelin-webyarn run dev:heliumYou can browse localhost:9000. Everytime refresh your browser, Zeppelin will rebuild your visualization and reload changes.5. Publish your visualizationOnce it's done, publish your visualization package using npm publish.That's it. With in an hour, your visualization will be available in Zeppelin's helium menu.See MoreCheck Helium Visualization: Transformation for more complex examples.",
"url": " /development/helium/writing_visualization_basic.html",
"group": "development/helium",
"excerpt": "Apache Zeppelin Visualization is a pluggable package that can be loaded/unloaded on runtime through Helium framework in Zeppelin. A Visualization is a javascript npm package and user can use them just like any other built-in visualization in a note."
}
,
"/development/helium/overview.html": {
"title": "Helium",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Helium OverviewWhat is Helium?Helium is a plugin system that can extend Zeppelin a lot. For example, you can write custom display system or install already published one in Heliun Online Registry. Currently, Helium supports 4 types of package.Helium Visualization: Adding a new chart typeHelium Spell: Adding new interpreter, display system running on browserHelium Application Helium Interpreter: Adding a new custom interpreter",
"url": " /development/helium/overview.html",
"group": "development/helium",
"excerpt": ""
}
,
"/development/helium/writing_application.html": {
"title": "Writing a new Helium Application",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Writing a new ApplicationWhat is Apache Zeppelin ApplicationApache Zeppelin Application is a package that runs on Interpreter process and displays it's output inside of the notebook. While application runs on Interpreter process, it's able to access resources provided by Interpreter through ResourcePool. Output is always rendered by AngularDisplaySystem. Therefore application provides all the possiblities of making interactive graphical application that uses data and processing power of any Interpreter.Make your own ApplicationWriting Application means extending org.apache.zeppelin.helium.Application. You can use your favorite IDE and language while Java class files are packaged into jar. Application class looks like/** * Constructor. Invoked when application is loaded */public Application(ApplicationContext context);/** * Invoked when there're (possible) updates in required resource set. * i.e. invoked after application load and after paragraph finishes. */public abstract void run(ResourceSet args);/** * Invoked before application unload. * Application is automatically unloaded with paragraph/notebook removal */public abstract void unload();You can check example applications under ./zeppelin-examples directory.Development modeIn the development mode, you can run your Application in your IDE as a normal java application and see the result inside of Zeppelin notebook.org.apache.zeppelin.helium.ZeppelinApplicationDevServer can run Zeppelin Application in development mode.// entry point for development modepublic static void main(String[] args) throws Exception { // add resources for development mode LocalResourcePool pool = new LocalResourcePool("dev"); pool.put("date", new Date()); // run application in devlopment mode with given resource // in this case, Clock.class.getName() will be the application class name org.apache.zeppelin.helium.ZeppelinApplicationDevServer devServer = new org.apache.zeppelin.helium.ZeppelinApplicationDevServer( Clock.class.getName(), pool.getAll()); // start development mode devServer.start(); devServer.join();}In the Zeppelin notebook, run %dev run will connect to application running in development mode.Package filePackage file is a json file that provides information about the application.Json file contains the following information{ "name" : "[organization].[name]", "description" : "Description", "artifact" : "groupId:artifactId:version", "className" : "your.package.name.YourApplicationClass", "resources" : [ ["resource.name", ":resource.class.name"], ["alternative.resource.name", ":alternative.class.name"] ], "icon" : "<i class='icon'></i>"}nameName is a string in [group].[name] format.[group] and [name] allow only [A-Za-z0-9_].Group is normally the name of an organization who creates this application.descriptionA short description about the applicationartifactLocation of the jar artifact."groupId:artifactId:version" will load artifact from maven repository.If jar exists in the local filesystem, absolute/relative can be used.e.g.When artifact exists in Maven repositoryartifact: "org.apache.zeppelin:zeppelin-examples:0.6.0"When artifact exists in the local filesystemartifact: "zeppelin-example/target/zeppelin-example-0.6.0.jar"classNameEntry point. Class that extends org.apache.zeppelin.helium.ApplicationresourcesTwo dimensional array that defines required resources by name or by className. Helium Application launcher will compare resources in the ResourcePool with the information in this field and suggest application only when all required resources are available in the ResourcePool.Resouce name is a string which will be compared with the name of objects in the ResourcePool. className is a string with ":" prepended, which will be compared with className of the objects in the ResourcePool.Application may require two or more resources. Required resources can be listed inside of the json array. For example, if the application requires object "name1", "name2" and "className1" type of object to run, resources field can beresources: [ [ "name1", "name2", ":className1", ...]]If Application can handle alternative combination of required resources, alternative set can be listed as below.resources: [ [ "name", ":className"], [ "altName", ":altClassName1"], ...]Easier way to understand this scheme isresources: [ [ 'resource' AND 'resource' AND ... ] OR [ 'resource' AND 'resource' AND ... ] OR ...]iconIcon to be used on the application button. String in this field will be rendered as a HTML tag.e.g.icon: "<i class='fa fa-clock-o'></i>"",
"url": " /development/helium/writing_application.html",
"group": "development/helium",
"excerpt": "Apache Zeppelin Application is a package that runs on Interpreter process and displays it's output inside of the notebook. Make your own Application in Apache Zeppelin is quite easy."
}
,
"/development/writing_zeppelin_interpreter.html": {
"title": "Writing a New Interpreter",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Writing a New InterpreterWhat is Apache Zeppelin InterpreterApache Zeppelin Interpreter is a language backend. For example to use scala code in Zeppelin, you need a scala interpreter.Every Interpreters belongs to an InterpreterGroup.Interpreters in the same InterpreterGroup can reference each other. For example, SparkSqlInterpreter can reference SparkInterpreter to get SparkContext from it while they're in the same group.InterpreterSetting is configuration of a given InterpreterGroup and a unit of start/stop interpreter.All Interpreters in the same InterpreterSetting are launched in a single, separate JVM process. The Interpreter communicates with Zeppelin engine via Thrift.In 'Separate Interpreter(scoped / isolated) for each note' mode which you can see at the Interpreter Setting menu when you create a new interpreter, new interpreter instance will be created per note. But it still runs on the same JVM while they're in the same InterpreterSettings.Make your own InterpreterCreating a new interpreter is quite simple. Just extend org.apache.zeppelin.interpreter abstract class and implement some methods.For your interpreter project, you need to make interpreter-parent as your parent project and use plugin maven-enforcer-plugin, maven-dependency-plugin and maven-resources-plugin. Here's one sample pom.xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <artifactId>interpreter-parent</artifactId> <groupId>org.apache.zeppelin</groupId> <version>0.8.0-SNAPSHOT</version> <relativePath>../interpreter-parent</relativePath> </parent> ... <dependencies> <dependency> <groupId>org.apache.zeppelin</groupId> <artifactId>zeppelin-interpreter</artifactId> <version>${project.version}</version> <scope>provided</scope> </dependency> </dependencies> <build> <plugins> <plugin> <artifactId>maven-enforcer-plugin</artifactId> </plugin> <plugin> <artifactId>maven-dependency-plugin</artifactId> </plugin> <plugin> <artifactId>maven-resources-plugin</artifactId> </plugin> </plugins> </build></project>You should include org.apache.zeppelin:zeppelin-interpreter:[VERSION] as your interpreter's dependency in pom.xml. BesAnd you should put your jars under your interpreter directory with a specific directory name. Zeppelin server reads interpreter directories recursively and initializes interpreters including your own interpreter.There are three locations where you can store your interpreter group, name and other information. Zeppelin server tries to find the location below. Next, Zeppelin tries to find interpreter-setting.json in your interpreter jar.{ZEPPELIN_INTERPRETER_DIR}/{YOUR_OWN_INTERPRETER_DIR}/interpreter-setting.jsonHere is an example of interpreter-setting.json on your own interpreter.[ { "group": "your-group", "name": "your-name", "className": "your.own.interpreter.class", "properties": { "properties1": { "envName": null, "propertyName": "property.1.name", "defaultValue": "propertyDefaultValue", "description": "Property description", "type": "textarea" }, "properties2": { "envName": PROPERTIES_2, "propertyName": null, "defaultValue": "property2DefaultValue", "description": "Property 2 description", "type": "textarea" }, ... }, "editor": { "language": "your-syntax-highlight-language", "editOnDblClick": false, "completionKey": "TAB" } }, { ... }]Finally, Zeppelin uses static initialization with the following:static { Interpreter.register("MyInterpreterName", MyClassName.class.getName());}Static initialization is deprecated and will be supported until 0.6.0.The name will appear later in the interpreter name option box during the interpreter configuration process.The name of the interpreter is what you later write to identify a paragraph which should be interpreted using this interpreter.%MyInterpreterNamesome interpreter specific code...Editor setting for InterpreterYou can add editor object to interpreter-setting.json file to specify paragraph editor settings.LanguageIf the interpreter uses a specific programming language (like Scala, Python, SQL), it is generally recommended to add a syntax highlighting supported for that to the note paragraph editor.To check out the list of languages supported, see the mode-*.js files under zeppelin-web/bower_components/ace-builds/src-noconflict or from github.com/ajaxorg/ace-builds.If you want to add a new set of syntax highlighting, Add the mode-*.js file to zeppelin-web/bower.json (when built, zeppelin-web/src/index.html will be changed automatically).Add language field to editor object. Note that if you don't specify language field, your interpreter will use plain text mode for syntax highlighting. Let's say you want to set your language to java, then add:"editor": { "language": "java"}Edit on double clickIf your interpreter uses mark-up language such as markdown or HTML, set editOnDblClick to true so that text editor opens on pargraph double click and closes on paragraph run. Otherwise set it to false."editor": { "editOnDblClick": false}Completion key (Optional)By default, Ctrl+dot(.) brings autocompletion list in the editor.Through completionKey, each interpreter can configure autocompletion key.Currently TAB is only available option."editor": { "completionKey": "TAB"}Install your interpreter binaryOnce you have built your interpreter, you can place it under the interpreter directory with all its dependencies.[ZEPPELIN_HOME]/interpreter/[INTERPRETER_NAME]/Configure your interpreterTo configure your interpreter you need to follow these steps:Add your interpreter class name to the zeppelin.interpreters property in conf/zeppelin-site.xml.Property value is comma separated [INTERPRETER_CLASS_NAME].For example,<property><name>zeppelin.interpreters</name><value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter,com.me.MyNewInterpreter</value></property>Add your interpreter to the default configuration which is used when there is no zeppelin-site.xml.Start Zeppelin by running ./bin/zeppelin-daemon.sh start.In the interpreter page, click the +Create button and configure your interpreter properties.Now you are done and ready to use your interpreter.Note : Interpreters released with zeppelin have a default configuration which is used when there is no conf/zeppelin-site.xml.Use your interpreter0.5.0Inside of a note, %[INTERPRETER_NAME] directive will call your interpreter.Note that the first interpreter configuration in zeppelin.interpreters will be the default one.For example,%myintpval a = "My interpreter"println(a)0.6.0 and laterInside of a note, %[INTERPRETER_GROUP].[INTERPRETER_NAME] directive will call your interpreter.You can omit either [INTERPRETER_GROUP] or [INTERPRETER_NAME]. If you omit [INTERPRETER_NAME], then first available interpreter will be selected in the [INTERPRETER_GROUP].Likewise, if you skip [INTERPRETER_GROUP], then [INTERPRETER_NAME] will be chosen from default interpreter group.For example, if you have two interpreter myintp1 and myintp2 in group mygrp, you can call myintp1 like%mygrp.myintp1codes for myintp1and you can call myintp2 like%mygrp.myintp2codes for myintp2If you omit your interpreter name, it'll select first available interpreter in the group ( myintp1 ).%mygrpcodes for myintp1You can only omit your interpreter group when your interpreter group is selected as a default group.%myintp2codes for myintp2ExamplesCheckout some interpreters released with Zeppelin by default.sparkmarkdownshelljdbcContributing a new Interpreter to Zeppelin releasesWe welcome contribution to a new interpreter. Please follow these few steps:First, check out the general contribution guide here.Follow the steps in Make your own Interpreter section and Editor setting for Interpreter above.Add your interpreter as in the Configure your interpreter section above; also add it to the example template zeppelin-site.xml.template.Add tests! They are run by Travis for all changes and it is important that they are self-contained.Include your interpreter as a module in pom.xml.Add documentation on how to use your interpreter under docs/interpreter/. Follow the Markdown style as this example. Make sure you list config settings and provide working examples on using your interpreter in code boxes in Markdown. Link to images as appropriate (images should go to docs/assets/themes/zeppelin/img/docs-img/). And add a link to your documentation in the navigation menu (docs/_includes/themes/zeppelin/_navigation.html).Most importantly, ensure licenses of the transitive closure of all dependencies are list in license file.Commit your changes and open a Pull Request on the project Mirror on GitHub; check to make sure Travis CI build is passing.",
"url": " /development/writing_zeppelin_interpreter.html",
"group": "development",
"excerpt": "Apache Zeppelin Interpreter is a language backend. Every Interpreters belongs to an InterpreterGroup. Interpreters in the same InterpreterGroup can reference each other."
}
,
"/development/contribution/how_to_contribute_website.html": {
"title": "Contributing to Apache Zeppelin (Website)",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Contributing to Apache Zeppelin ( Website )This page will give you an overview of how to build and contribute to the documentation of Apache Zeppelin.The online documentation at zeppelin.apache.org is also generated from the files found here.NOTE : Apache Zeppelin is an Apache2 License Software.Any contributions to Zeppelin (Source code, Documents, Image, Website) means you agree with license all your contributions as Apache2 License.Getting the source codeFirst of all, you need Zeppelin source code. The official location of Zeppelin is http://git.apache.org/zeppelin.git.Documentation website is hosted in 'master' branch under /docs/ dir.git accessFirst of all, you need the website source code. The official location of mirror for Zeppelin is http://git.apache.org/zeppelin.git.Get the source code on your development machine using git.git clone git://git.apache.org/zeppelin.gitcd docsApache Zeppelin follows Fork & Pull as a source control workflow.If you want to not only build Zeppelin but also make any changes, then you need to fork Zeppelin github mirror repository and make a pull request.BuildYou'll need to install some prerequisites to build the code. Please check Build documentation section in docs/README.md.Run website in development modeWhile you're modifying website, you might want to see preview of it. Please check Run website section in docs/README.md.Then you'll be able to access it on http://localhost:4000 with your web browser.Making a Pull RequestWhen you are ready, just make a pull-request.Alternative wayYou can directly edit .md files in /docs/ directory at the web interface of github and make pull-request immediately.Stay involvedContributors should join the Zeppelin mailing lists.dev@zeppelin.apache.org is for people who want to contribute code to Zeppelin. subscribe, unsubscribe, archivesIf you have any issues, create a ticket in JIRA.",
"url": " /development/contribution/how_to_contribute_website.html",
"group": "development/contribution",
"excerpt": "How can you contribute to Apache Zeppelin project website? This document covers from building Zeppelin documentation site to making a pull request on Github."
}
,
"/development/contribution/how_to_contribute_code.html": {
"title": "Contributing to Apache Zeppelin (Code)",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Contributing to Apache Zeppelin ( Code )NOTE : Apache Zeppelin is an Apache2 License Software.Any contributions to Zeppelin (Source code, Documents, Image, Website) means you agree with license all your contributions as Apache2 License.Setting upHere are some tools you will need to build and test Zeppelin.Software Configuration Management ( SCM )Since Zeppelin uses Git for it's SCM system, you need git client installed in your development machine.Integrated Development Environment ( IDE )You are free to use whatever IDE you prefer, or your favorite command line editor.Build ToolsTo build the code, installOracle Java 7Apache MavenGetting the source codeFirst of all, you need Zeppelin source code. The official location of Zeppelin is http://git.apache.org/zeppelin.git.git accessGet the source code on your development machine using git.git clone git://git.apache.org/zeppelin.git zeppelinYou may also want to develop against a specific branch. For example, for branch-0.5.6git clone -b branch-0.5.6 git://git.apache.org/zeppelin.git zeppelinApache Zeppelin follows Fork & Pull as a source control workflow.If you want to not only build Zeppelin but also make any changes, then you need to fork Zeppelin github mirror repository and make a pull request.Before making a pull request, please take a look Contribution Guidelines.Buildmvn installTo skip testmvn install -DskipTestsTo build with specific spark / hadoop versionmvn install -Dspark.version=x.x.x -Dhadoop.version=x.x.xFor the further Run Zeppelin server in development modeOption 1 - Command LineCopy the conf/zeppelin-site.xml.template to zeppelin-server/src/main/resources/zeppelin-site.xml and change the configurations in this file if requiredRun the following commandcd zeppelin-serverHADOOP_HOME=YOUR_HADOOP_HOME JAVA_HOME=YOUR_JAVA_HOME mvn exec:java -Dexec.mainClass="org.apache.zeppelin.server.ZeppelinServer" -Dexec.args=""Option 2 - Daemon ScriptNote: Make sure you first run mvn clean install -DskipTestsin your zeppelin root directory, otherwise your server build will fail to find the required dependencies in the local repro.or use daemon scriptbin/zeppelin-daemon startServer will be run on http://localhost:8080.Option 3 - IDECopy the conf/zeppelin-site.xml.template to zeppelin-server/src/main/resources/zeppelin-site.xml and change the configurations in this file if requiredZeppelinServer.java Main classGenerating Thrift CodeSome portions of the Zeppelin code are generated by Thrift. For most Zeppelin changes, you don't need to worry about this. But if you modify any of the Thrift IDL files (e.g. zeppelin-interpreter/src/main/thrift/*.thrift), then you also need to regenerate these files and submit their updated version as part of your patch.To regenerate the code, install thrift-0.9.2 and then run the following command to generate thrift code.cd <zeppelin_home>/zeppelin-interpreter/src/main/thrift./genthrift.shRun Selenium testZeppelin has set of integration tests using Selenium. To run these test, first build and run Zeppelin and make sure Zeppelin is running on port 8080. Then you can run test using following commandTEST_SELENIUM=true mvn test -Dtest=[TEST_NAME] -DfailIfNoTests=false -pl 'zeppelin-interpreter,zeppelin-zengine,zeppelin-server'For example, to run ParagraphActionIT,TEST_SELENIUM=true mvn test -Dtest=ParagraphActionsIT -DfailIfNoTests=false -pl 'zeppelin-interpreter,zeppelin-zengine,zeppelin-server'You'll need Firefox web browser installed in your development environment. While CI server uses Firefox 31.0 to run selenium test, it is good idea to install the same version (disable auto update to keep the version).Where to StartYou can find issues for beginner & newbieStay involvedContributors should join the Zeppelin mailing lists.dev@zeppelin.apache.org is for people who want to contribute code to Zeppelin. subscribe, unsubscribe, archivesIf you have any issues, create a ticket in JIRA.",
"url": " /development/contribution/how_to_contribute_code.html",
"group": "development/contribution",
"excerpt": "How can you contribute to Apache Zeppelin project? This document covers from setting up your develop environment to making a pull request on Github."
}
,
"/development/contribution/useful_developer_tools.html": {
"title": "Useful Developer Tools",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Useful Developer ToolsDeveloping zeppelin-webCheck zeppelin-web: Local Development.ToolsSVM: Scala Version Managersvm would be useful when changing scala version frequently.JDK change script: OSXthis script would be helpful when changing JDK version frequently.function setjdk() { if [ $# -ne 0 ]; then # written based on OSX. # use diffrent base path for other OS removeFromPath '/System/Library/Frameworks/JavaVM.framework/Home/bin' if [ -n "${JAVA_HOME+x}" ]; then removeFromPath $JAVA_HOME fi export JAVA_HOME=`/usr/libexec/java_home -v $@` export PATH=$JAVA_HOME/bin:$PATH fi}function removeFromPath() { export PATH=$(echo $PATH | sed -E -e "s;:$1;;" -e "s;$1:?;;")}you can use this function like setjdk 1.8 / setjdk 1.7Building Submodules Selectively# build `zeppelin-web` onlymvn clean -pl 'zeppelin-web' package -DskipTests;# build `zeppelin-server` and its dependencies onlymvn clean package -pl 'spark,spark-dependencies,python,markdown,zeppelin-server' --am -DskipTests# build spark related modules with default profiles: scala 2.10 mvn clean package -pl 'spark,spark-dependencies,zeppelin-server' --am -DskipTests# build spark related modules with profiles: scala 2.11, spark 2.1 hadoop 2.7 ./dev/change_scala_version.sh 2.11mvn clean package -Pspark-2.1 -Phadoop-2.7 -Pscala-2.11 -pl 'spark,spark-dependencies,zeppelin-server' --am -DskipTests# build `zeppelin-server` and `markdown` with dependenciesmvn clean package -pl 'markdown,zeppelin-server' --am -DskipTestsRunning Individual Tests# run the `HeliumBundleFactoryTest` test classmvn test -pl 'zeppelin-server' --am -DfailIfNoTests=false -Dtest=HeliumBundleFactoryTestRunning Selenium TestsMake sure that Zeppelin instance is started to execute integration tests (= selenium tests).# run the `SparkParagraphIT` test classTEST_SELENIUM="true" mvn test -pl 'zeppelin-server' --am -DfailIfNoTests=false -Dtest=SparkParagraphIT# run the `testSqlSpark` test function only in the `SparkParagraphIT` class# but note that, some test might be dependent on the previous testsTEST_SELENIUM="true" mvn test -pl 'zeppelin-server' --am -DfailIfNoTests=false -Dtest=SparkParagraphIT#testSqlSpark",
"url": " /development/contribution/useful_developer_tools.html",
"group": "development/contribution",
"excerpt": ""
}
,
"/setup/operation/proxy_setting.html": {
"title": "Proxy Setting in Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Proxy Setting ## How to Configure Proxies?Set `http_proxy` and `https_proxy` env variables. (See [more](https://wiki.archlinux.org/index.php/proxy_settings))Currently, Proxy is supported only for these features.- Helium: downloading `helium.json`, installing `npm`, `node`, `yarn`",
"url": " /setup/operation/proxy_setting.html",
"group": "security",
"excerpt": "Apache Zeppelin supports Helium plugins which fetch required installer packages from remote registry/repositories"
}
,
"/setup/operation/trouble_shooting.html": {
"title": "Trouble Shooting",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Troubleshooting ",
"url": " /setup/operation/trouble_shooting.html",
"group": "setup/operation",
"excerpt": ""
}
,
"/setup/operation/configuration.html": {
"title": "Apache Zeppelin Configuration",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Apache Zeppelin Configuration## Zeppelin PropertiesThere are two locations you can configure Apache Zeppelin.* **Environment variables** can be defined `conf/zeppelin-env.sh`(`confzeppelin-env.cmd` for Windows).* **Java properties** can be defined in `conf/zeppelin-site.xml`.If both are defined, then the **environment variables** will take priority.> Mouse hover on each property and click then you can get a link for that. zeppelin-env.sh zeppelin-site.xml Default value Description ZEPPELIN_ADDR zeppelin.server.addr 127.0.0.1 Zeppelin server binding address ZEPPELIN_PORT zeppelin.server.port 8080 Zeppelin server port Note: Please make sure you're not using the same port with Zeppelin web application development port (default: 9000). ZEPPELIN_SSL_PORT zeppelin.server.ssl.port 8443 Zeppelin Server ssl port (used when ssl environment/property is set to true) ZEPPELIN_JMX_ENABLE N/A Enable JMX by defining "true" ZEPPELIN_JMX_PORT N/A 9996 Port number which JMX uses ZEPPELIN_MEM N/A -Xmx1024m -XX:MaxPermSize=512m JVM mem options ZEPPELIN_INTP_MEM N/A ZEPPELIN_MEM JVM mem options for interpreter process ZEPPELIN_JAVA_OPTS N/A JVM options ZEPPELIN_ALLOWED_ORIGINS zeppelin.server.allowed.origins * Enables a way to specify a ',' separated list of allowed origins for REST and websockets. e.g. http://localhost:8080 ZEPPELIN_CREDENTIALS_PERSIST zeppelin.credentials.persist true Persist credentials on a JSON file (credentials.json) ZEPPELIN_CREDENTIALS_ENCRYPT_KEY zeppelin.credentials.encryptKey If provided, encrypt passwords on the credentials.json file (passwords will be stored as plain-text otherwise N/A zeppelin.anonymous.allowed true The anonymous user is allowed by default. ZEPPELIN_SERVER_CONTEXT_PATH zeppelin.server.context.path / Context path of the web application ZEPPELIN_SSL zeppelin.ssl false ZEPPELIN_SSL_CLIENT_AUTH zeppelin.ssl.client.auth false ZEPPELIN_SSL_KEYSTORE_PATH zeppelin.ssl.keystore.path keystore ZEPPELIN_SSL_KEYSTORE_TYPE zeppelin.ssl.keystore.type JKS ZEPPELIN_SSL_KEYSTORE_PASSWORD zeppelin.ssl.keystore.password ZEPPELIN_SSL_KEY_MANAGER_PASSWORD zeppelin.ssl.key.manager.password ZEPPELIN_SSL_TRUSTSTORE_PATH zeppelin.ssl.truststore.path ZEPPELIN_SSL_TRUSTSTORE_TYPE zeppelin.ssl.truststore.type ZEPPELIN_SSL_TRUSTSTORE_PASSWORD zeppelin.ssl.truststore.password ZEPPELIN_NOTEBOOK_HOMESCREEN zeppelin.notebook.homescreen Display note IDs on the Apache Zeppelin homescreen e.g. 2A94M5J1Z ZEPPELIN_NOTEBOOK_HOMESCREEN_HIDE zeppelin.notebook.homescreen.hide false Hide the note ID set by ZEPPELIN_NOTEBOOK_HOMESCREEN on the Apache Zeppelin homescreen. For the further information, please read Customize your Zeppelin homepage. ZEPPELIN_WAR_TEMPDIR zeppelin.war.tempdir webapps Location of the jetty temporary directory ZEPPELIN_NOTEBOOK_DIR zeppelin.notebook.dir notebook The root directory where notebook directories are saved ZEPPELIN_NOTEBOOK_S3_BUCKET zeppelin.notebook.s3.bucket zeppelin S3 Bucket where notebook files will be saved ZEPPELIN_NOTEBOOK_S3_USER zeppelin.notebook.s3.user user User name of an S3 buckete.g. bucket/user/notebook/2A94M5J1Z/note.json ZEPPELIN_NOTEBOOK_S3_ENDPOINT zeppelin.notebook.s3.endpoint s3.amazonaws.com Endpoint for the bucket ZEPPELIN_NOTEBOOK_S3_KMS_KEY_ID zeppelin.notebook.s3.kmsKeyID AWS KMS Key ID to use for encrypting data in S3 (optional) ZEPPELIN_NOTEBOOK_S3_EMP zeppelin.notebook.s3.encryptionMaterialsProvider Class name of a custom S3 encryption materials provider implementation to use for encrypting data in S3 (optional) ZEPPELIN_NOTEBOOK_S3_SSE zeppelin.notebook.s3.sse false Save notebooks to S3 with server-side encryption enabled ZEPPELIN_NOTEBOOK_S3_SIGNEROVERRIDE zeppelin.notebook.s3.signerOverride Optional override to control which signature algorithm should be used to sign AWS requests ZEPPELIN_NOTEBOOK_AZURE_CONNECTION_STRING zeppelin.notebook.azure.connectionString The Azure storage account connection stringe.g. DefaultEndpointsProtocol=https;AccountName=<accountName>;AccountKey=<accountKey> ZEPPELIN_NOTEBOOK_AZURE_SHARE zeppelin.notebook.azure.share zeppelin Azure Share where the notebook files will be saved ZEPPELIN_NOTEBOOK_AZURE_USER zeppelin.notebook.azure.user user Optional user name of an Azure file sharee.g. share/user/notebook/2A94M5J1Z/note.json ZEPPELIN_NOTEBOOK_STORAGE zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.GitNotebookRepo Comma separated list of notebook storage locations ZEPPELIN_NOTEBOOK_ONE_WAY_SYNC zeppelin.notebook.one.way.sync false If there are multiple notebook storage locations, should we treat the first one as the only source of truth? ZEPPELIN_NOTEBOOK_PUBLIC zeppelin.notebook.public true Make notebook public (set only owners) by default when created/imported. If set to false will add user to readers and writers as well, making it private and invisible to other users unless permissions are granted. ZEPPELIN_INTERPRETERS zeppelin.interpreters org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter, ... Comma separated interpreter configurations [Class] NOTE: This property is deprecated since Zeppelin-0.6.0 and will not be supported from Zeppelin-0.7.0. ZEPPELIN_INTERPRETER_DIR zeppelin.interpreter.dir interpreter Interpreter directory ZEPPELIN_INTERPRETER_DEP_MVNREPO zeppelin.interpreter.dep.mvnRepo http://repo1.maven.org/maven2/ Remote principal repository for interpreter's additional dependency loading ZEPPELIN_INTERPRETER_OUTPUT_LIMIT zeppelin.interpreter.output.limit 102400 Output message from interpreter exceeding the limit will be truncated ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT zeppelin.interpreter.connect.timeout 30000 Output message from interpreter exceeding the limit will be truncated ZEPPELIN_DEP_LOCALREPO zeppelin.dep.localrepo local-repo Local repository for dependency loader.ex)visualiztion modules of npm. ZEPPELIN_HELIUM_NODE_INSTALLER_URL zeppelin.helium.node.installer.url https://nodejs.org/dist/ Remote Node installer url for Helium dependency loader ZEPPELIN_HELIUM_NPM_INSTALLER_URL zeppelin.helium.npm.installer.url http://registry.npmjs.org/ Remote Npm installer url for Helium dependency loader ZEPPELIN_HELIUM_YARNPKG_INSTALLER_URL zeppelin.helium.yarnpkg.installer.url https://github.com/yarnpkg/yarn/releases/download/ Remote Yarn package installer url for Helium dependency loader ZEPPELIN_WEBSOCKET_MAX_TEXT_MESSAGE_SIZE zeppelin.websocket.max.text.message.size 1024000 Size(in characters) of the maximum text message that can be received by websocket. ZEPPELIN_SERVER_DEFAULT_DIR_ALLOWED zeppelin.server.default.dir.allowed false Enable directory listings on server. ZEPPELIN_NOTEBOOK_GIT_REMOTE_URL zeppelin.notebook.git.remote.url GitHub's repository URL. It could be either the HTTP URL or the SSH URL. For example git@github.com:apache/zeppelin.git ZEPPELIN_NOTEBOOK_GIT_REMOTE_USERNAME zeppelin.notebook.git.remote.username token GitHub username. By default it is `token` to use GitHub's API ZEPPELIN_NOTEBOOK_GIT_REMOTE_ACCESS_TOKEN zeppelin.notebook.git.remote.access-token token GitHub access token to use GitHub's API. If username/password combination is used and not GitHub API, then this value is the password ZEPPELIN_NOTEBOOK_GIT_REMOTE_ORIGIN zeppelin.notebook.git.remote.origin token GitHub remote name. Default is `origin` ## SSL ConfigurationEnabling SSL requires a few configuration changes. First, you need to create certificates and then update necessary configurations to enable server side SSL and/or client side certificate authentication.### Creating and configuring the CertificatesInformation how about to generate certificates and a keystore can be found [here](https://wiki.eclipse.org/Jetty/Howto/Configure_SSL).A condensed example can be found in the top answer to this [StackOverflow post](http://stackoverflow.com/questions/4008837/configure-ssl-on-jetty).The keystore holds the private key and certificate on the server end. The trustore holds the trusted client certificates. Be sure that the path and password for these two stores are correctly configured in the password fields below. They can be obfuscated using the Jetty password tool. After Maven pulls in all the dependency to build Zeppelin, one of the Jetty jars contain the Password tool. Invoke this command from the Zeppelin home build directory with the appropriate version, user, and password.```bashjava -cp ./zeppelin-server/target/lib/jetty-all-server-.jar org.eclipse.jetty.util.security.Password ```If you are using a self-signed, a certificate signed by an untrusted CA, or if client authentication is enabled, then the client must have a browser create exceptions for both the normal HTTPS port and WebSocket port. This can by done by trying to establish an HTTPS connection to both ports in a browser (e.g. if the ports are 443 and 8443, then visit https://127.0.0.1:443 and https://127.0.0.1:8443). This step can be skipped if the server certificate is signed by a trusted CA and client auth is disabled.### Configuring server side SSLThe following properties needs to be updated in the `zeppelin-site.xml` in order to enable server side SSL.```xml zeppelin.server.ssl.port 8443 Server ssl port. (used when ssl property is set to true) zeppelin.ssl true Should SSL be used by the servers? zeppelin.ssl.keystore.path keystore Path to keystore relative to Zeppelin configuration directory zeppelin.ssl.keystore.type JKS The format of the given keystore (e.g. JKS or PKCS12) zeppelin.ssl.keystore.password change me Keystore password. Can be obfuscated by the Jetty Password tool zeppelin.ssl.key.manager.password change me Key Manager password. Defaults to keystore password. Can be obfuscated.```### Enabling client side certificate authenticationThe following properties needs to be updated in the `zeppelin-site.xml` in order to enable client side certificate authentication.```xml zeppelin.server.ssl.port 8443 Server ssl port. (used when ssl property is set to true) zeppelin.ssl.client.auth true Should client authentication be used for SSL connections? zeppelin.ssl.truststore.path truststore Path to truststore relative to Zeppelin configuration directory. Defaults to the keystore path zeppelin.ssl.truststore.type JKS The format of the given truststore (e.g. JKS or PKCS12). Defaults to the same type as the keystore type zeppelin.ssl.truststore.password change me Truststore password. Can be obfuscated by the Jetty Password tool. Defaults to the keystore password```### Storing user credentialsIn order to avoid having to re-enter credentials every time you restart/redeploy Zeppelin, you can store the user credentials. Zeppelin supports this via the ZEPPELIN_CREDENTIALS_PERSIST configuration.Please notice that passwords will be stored in *plain text* by default. To encrypt the passwords, use the ZEPPELIN_CREDENTIALS_ENCRYPT_KEY config variable. This will encrypt passwords using the AES-128 algorithm.You can generate an appropriate encryption key any way you'd like - for instance, by using the openssl tool:```bashopenssl enc -aes-128-cbc -k secret -P -md sha1```*Important*: storing your encryption key in a configuration file is _not advised_. Depending on your environment security needs, you may want to consider utilizing a credentials server, storing the ZEPPELIN_CREDENTIALS_ENCRYPT_KEY as an OS env variable, or any other approach that would not colocate the encryption key and the encrypted content (the credentials.json file).### Obfuscating Passwords using the Jetty Password ToolSecurity best practices advise to not use plain text passwords and Jetty provides a password tool to help obfuscating the passwords used to access the KeyStore and TrustStore.The Password tool documentation can be found [here](http://www.eclipse.org/jetty/documentation/current/configuring-security-secure-passwords.html).After using the tool:```bashjava -cp $ZEPPELIN_HOME/zeppelin-server/target/lib/jetty-util-9.2.15.v20160210.jar org.eclipse.jetty.util.security.Password password2016-12-15 10:46:47.931:INFO::main: Logging initialized @101mspasswordOBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1vMD5:5f4dcc3b5aa765d61d8327deb882cf99```update your configuration with the obfuscated password :```xml zeppelin.ssl.keystore.password OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v Keystore password. Can be obfuscated by the Jetty Password tool```### Create GitHub Access TokenWhen using GitHub to track notebooks, one can use GitHub's API for authentication. To create an access token, please use the following link https://github.com/settings/tokens.The value of the access token generated is set in the `zeppelin.notebook.git.remote.access-token` property.**Note:** After updating these configurations, Zeppelin server needs to be restarted.",
"url": " /setup/operation/configuration.html",
"group": "setup/operation",
"excerpt": "This page will guide you to configure Apache Zeppelin using either environment variables or Java properties. Also, you can configure SSL for Zeppelin."
}
,
"/setup/operation/upgrading.html": {
"title": "Manual Zeppelin version upgrade procedure",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Manual upgrade procedure for ZeppelinBasically, newer version of Zeppelin works with previous version notebook directory and configurations.So, copying `notebook` and `conf` directory should be enough.## Instructions1. Stop Zeppelin: `bin/zeppelin-daemon.sh stop`2. Copy your `notebook` and `conf` directory into a backup directory3. Download newer version of Zeppelin and Install. See [Install Guide](../../quickstart/install.html#install).4. Copy backup `notebook` and `conf` directory into newer version of Zeppelin `notebook` and `conf` directory5. Start Zeppelin: `bin/zeppelin-daemon.sh start`## Migration Guide### Breaking changes in 0.8.xFrom 0.8, Zeppelin has a new type of permission - [Runners](http://zeppelin.apache.org/docs/0.8.0/setup/security/notebook_authorization.html#authorization-setting)As Runners list is empty in note so everybody can view note although Readers list is not empty. To set all your "writers" to "runners":1. Copy `notebook` and `conf` directories to 0.8.0,2. Move directory **docs/assets/themes/zeppelin/note/FixReaders** to new `notebook` directory,3. Start the new Zeppelin and run note **System/Migrate from 0.7**.### Upgrading from Zeppelin 0.8.1 (and before) to 0.8.2 (and later) - From 0.8.2, Zeppelin server bind `127.0.0.1` by default instead of `0.0.0.0`. Configure `zeppelin.server.addr` property or `ZEPPELIN_ADDR` env variable to change. ### Upgrading from Zeppelin 0.7 to 0.8 - From 0.8, we recommend to use `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` instead of `zeppelin.pyspark.python` as `zeppelin.pyspark.python` only effects driver. You can use `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` as using them in spark. - From 0.8, depending on your device, the keyboard shortcut `Ctrl-L` or `Command-L` which goes to the line somewhere user wants is not supported. ### Upgrading from Zeppelin 0.6 to 0.7 - From 0.7, we don't use `ZEPPELIN_JAVA_OPTS` as default value of `ZEPPELIN_INTP_JAVA_OPTS` and also the same for `ZEPPELIN_MEM`/`ZEPPELIN_INTP_MEM`. If user want to configure the jvm opts of interpreter process, please set `ZEPPELIN_INTP_JAVA_OPTS` and `ZEPPELIN_INTP_MEM` explicitly. If you don't set `ZEPPELIN_INTP_MEM`, Zeppelin will set it to `-Xms1024m -Xmx1024m -XX:MaxPermSize=512m` by default. - Mapping from `%jdbc(prefix)` to `%prefix` is no longer available. Instead, you can use %[interpreter alias] with multiple interpreter setttings on GUI. - Usage of `ZEPPELIN_PORT` is not supported in ssl mode. Instead use `ZEPPELIN_SSL_PORT` to configure the ssl port. Value from `ZEPPELIN_PORT` is used only when `ZEPPELIN_SSL` is set to `false`. - The support on Spark 1.1.x to 1.3.x is deprecated. - From 0.7, we uses `pegdown` as the `markdown.parser.type` option for the `%md` interpreter. Rendered markdown might be different from what you expected - From 0.7 note.json format has been changed to support multiple outputs in a paragraph. Zeppelin will automatically convert old format to new format. 0.6 or lower version can read new note.json format but output will not be displayed. For the detail, see [ZEPPELIN-212](http://issues.apache.org/jira/browse/ZEPPELIN-212) and [pull request](https://github.com/apache/zeppelin/pull/1658). - From 0.7 note storage layer will utilize `GitNotebookRepo` by default instead of `VFSNotebookRepo` storage layer, which is an extension of latter one with versioning capabilities on top of it.",
"url": " /setup/operation/upgrading.html",
"group": "setup/operation",
"excerpt": "This document will guide you through a procedure of manual upgrade your Apache Zeppelin instance to a newer version. Apache Zeppelin keeps backward compatibility for the notebook file format."
}
,
"/setup/basics/how_to_build.html": {
"title": "How to Build Zeppelin from source",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}## How to Build Zeppelin from Source#### 0. RequirementsIf you want to build from source, you must first install the following dependencies: Name Value Git (Any Version) Maven 3.1.x or higher JDK 1.7 If you haven't installed Git and Maven yet, check the [Build requirements](#build-requirements) section and follow the step by step instructions from there.#### 1. Clone the Apache Zeppelin repository```bashgit clone https://github.com/apache/zeppelin.git```#### 2. Build sourceYou can build Zeppelin with following maven command:```bashmvn clean package -DskipTests [Options]```If you're unsure about the options, use the same commands that creates official binary package.```bash# update all pom.xml to use scala 2.11./dev/change_scala_version.sh 2.11# build zeppelin with all interpreters and include latest version of Apache spark support for local mode.mvn clean package -DskipTests -Pspark-2.0 -Phadoop-2.4 -Pr -Pscala-2.11```#### 3. DoneYou can directly start Zeppelin by running after successful build:```bash./bin/zeppelin-daemon.sh start```Check [build-profiles](#build-profiles) section for further build options.If you are behind proxy, follow instructions in [Proxy setting](#proxy-setting-optional) section.If you're interested in contribution, please check [Contributing to Apache Zeppelin (Code)](../../development/contribution/how_to_contribute_code.html) and [Contributing to Apache Zeppelin (Website)](../../development/contribution/how_to_contribute_website.html).### Build profiles#### Spark InterpreterTo build with a specific Spark version, Hadoop version or specific features, define one or more of the following profiles and options:##### `-Pspark-[version]`Set spark major versionAvailable profiles are```-Pspark-2.1-Pspark-2.0-Pspark-1.6-Pspark-1.5-Pspark-1.4-Pcassandra-spark-1.5-Pcassandra-spark-1.4-Pcassandra-spark-1.3-Pcassandra-spark-1.2-Pcassandra-spark-1.1```minor version can be adjusted by `-Dspark.version=x.x.x`##### `-Phadoop-[version]`set hadoop major versionAvailable profiles are```-Phadoop-0.23-Phadoop-1-Phadoop-2.2-Phadoop-2.3-Phadoop-2.4-Phadoop-2.6-Phadoop-2.7```minor version can be adjusted by `-Dhadoop.version=x.x.x`##### `-Pscala-[version] (optional)`set scala version (default 2.10)Available profiles are```-Pscala-2.10-Pscala-2.11```##### `-Pr` (optional)enable [R](https://www.r-project.org/) support with [SparkR](https://spark.apache.org/docs/latest/sparkr.html) integration.##### `-Pvendor-repo` (optional)enable 3rd party vendor repository (cloudera)##### `-Pmapr[version]` (optional)For the MapR Hadoop Distribution, these profiles will handle the Hadoop version. As MapR allows different versions of Spark to be installed, you should specify which version of Spark is installed on the cluster by adding a Spark profile (`-Pspark-1.6`, `-Pspark-2.0`, etc.) as needed.The correct Maven artifacts can be found for every version of MapR at http://doc.mapr.comAvailable profiles are```-Pmapr3-Pmapr40-Pmapr41-Pmapr50-Pmapr51```#### -Pexamples (optional)Build examples under zeppelin-examples directory### Build command examplesHere are some examples with several options:```bash# build with spark-2.1, scala-2.11./dev/change_scala_version.sh 2.11mvn clean package -Pspark-2.1 -Phadoop-2.4 -Pscala-2.11 -DskipTests# build with spark-2.0, scala-2.11./dev/change_scala_version.sh 2.11mvn clean package -Pspark-2.0 -Phadoop-2.4 -Pscala-2.11 -DskipTests# build with spark-1.6, scala-2.10mvn clean package -Pspark-1.6 -Phadoop-2.4 -DskipTests# spark-cassandra integrationmvn clean package -Pcassandra-spark-1.5 -Dhadoop.version=2.6.0 -Phadoop-2.6 -DskipTests -DskipTests# with CDHmvn clean package -Pspark-1.5 -Dhadoop.version=2.6.0-cdh5.5.0 -Phadoop-2.6 -Pvendor-repo -DskipTests# with MapRmvn clean package -Pspark-1.5 -Pmapr50 -DskipTests```Ignite Interpreter```bashmvn clean package -Dignite.version=1.9.0 -DskipTests```Scalding Interpreter```bashmvn clean package -Pscalding -DskipTests```### Optional configurationsHere are additional configurations that could be optionally tuned using the trailing `-D` option for maven commandsSpark package```bashspark.archive # default spark-${spark.version}spark.src.download.url # default http://d3kbcqa49mib13.cloudfront.net/${spark.archive}.tgzspark.bin.download.url # default http://d3kbcqa49mib13.cloudfront.net/${spark.archive}-bin-without-hadoop.tgz```Py4J package```bashpython.py4j.version # default 0.9.2pypi.repo.url # default https://pypi.python.org/packagespython.py4j.repo.folder # default /64/5c/01e13b68e8caafece40d549f232c9b5677ad1016071a48d04cc3895acaa3```final URL location for Py4J package will be produced as following:`${pypi.repo.url}${python.py4j.repo.folder}py4j-${python.py4j.version}.zip`Frontend Maven Plugin configurations```plugin.frontend.nodeDownloadRoot # default https://nodejs.org/dist/plugin.frontend.npmDownloadRoot # default http://registry.npmjs.org/npm/-/plugin.frontend.yarnDownloadRoot # default https://github.com/yarnpkg/yarn/releases/download/```## Build requirements### Install requirementsIf you don't have requirements prepared, install it.(The installation method may vary according to your environment, example is for Ubuntu.)```bashsudo apt-get updatesudo apt-get install gitsudo apt-get install openjdk-7-jdksudo apt-get install npmsudo apt-get install libfontconfigsudo apt-get install r-base-devsudo apt-get install r-cran-evaluate```### Install maven```bashwget http://www.eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gzsudo tar -zxf apache-maven-3.3.9-bin.tar.gz -C /usr/local/sudo ln -s /usr/local/apache-maven-3.3.9/bin/mvn /usr/local/bin/mvn```_Notes:_ - Ensure node is installed by running `node --version` - Ensure maven is running version 3.1.x or higher with `mvn -version` - Configure maven to use more memory than usual by `export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=1024m"`## Proxy setting (optional)If you're behind the proxy, you'll need to configure maven and npm to pass through it.First of all, configure maven in your `~/.m2/settings.xml`.```xml proxy-http true http localhost 3128 usr pwd --> localhost|127.0.0.1 proxy-https true https localhost 3128 usr pwd --> localhost|127.0.0.1 ```Then, next commands will configure npm.```bashnpm config set proxy http://localhost:3128npm config set https-proxy http://localhost:3128npm config set registry "http://registry.npmjs.org/"npm config set strict-ssl false```Configure git as well```bashgit config --global http.proxy http://localhost:3128git config --global https.proxy http://localhost:3128git config --global url."http://".insteadOf git://```To clean up, set `active false` in Maven `settings.xml` and run these commands.```bashnpm config rm proxynpm config rm https-proxygit config --global --unset http.proxygit config --global --unset https.proxygit config --global --unset url."http://".insteadOf```_Notes:_ - If you are behind NTLM proxy you can use [Cntlm Authentication Proxy](http://cntlm.sourceforge.net/). - Replace `localhost:3128` with the standard pattern `http://user:pwd@host:port`.## PackageTo package the final distribution including the compressed archive, run:```shmvn clean package -Pbuild-distr```To build a distribution with specific profiles, run:```shmvn clean package -Pbuild-distr -Pspark-1.5 -Phadoop-2.4```The profiles `-Pspark-1.5 -Phadoop-2.4` can be adjusted if you wish to build to a specific spark versions. The archive is generated under _`zeppelin-distribution/target`_ directory## Run end-to-end testsZeppelin comes with a set of end-to-end acceptance tests driving headless selenium browser```sh# assumes zeppelin-server running on localhost:8080 (use -Durl=.. to override)mvn verify# or take care of starting/stoping zeppelin-server from packaged zeppelin-distribuion/targetmvn verify -P using-packaged-distr```[](https://github.com/igrigorik/ga-beacon)",
"url": " /setup/basics/how_to_build.html",
"group": "setup/basics",
"excerpt": "How to build Zeppelin from source"
}
,
"/setup/basics/multi_user_support.html": {
"title": "Multi-user Support",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Multi-user Support This page describes about multi-user support.- multiple users login / logout using [Shiro Authentication](../security/shiro_authentication.html)- managing [Notebook Permission](../security/notebook_authorization.html)- how to setup [impersonation for interpreters](../../usage/interpreter/user_impersonation.html)- different contexts per user / note using [Interpreter Binding Mode](../../usage/interpreter/interpreter_binding_mode.html)- a paragraph in a notebook can be [Personalized](../../usage/other_features/personalized_mode.html) - propagates changes in notebooks through websocket in real-time",
"url": " /setup/basics/multi_user_support.html",
"group": "setup/basics",
"excerpt": ""
}
,
"/setup/security/http_security_headers.html": {
"title": "Setting up HTTP Response Headers",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Setting up HTTP Response Headers for Zeppelin Apache Zeppelin can be configured to include HTTP Headers which aids in preventing Cross Site Scripting (XSS), Cross-Frame Scripting (XFS) and also enforces HTTP Strict Transport Security. Apache Zeppelin also has configuration available to set the Application Server Version to desired value.## Setting up HTTP Strict Transport Security (HSTS) Response HeaderEnabling HSTS Response Header prevents Man-in-the-middle attacks by automatically redirecting HTTP requests to HTTPS when Zeppelin Server is running on SSL. Read on how to configure SSL for Zeppelin [here] (../operation/configuration.html). Even if web page contains any resource which gets served over HTTP or any HTTP links, it will automatically be redirected to HTTPS for the target domain. It also prevents MITM attack by not allowing User to override the invalid certificate message, when Attacker presents invalid SSL certificate to the User. The following property needs to be updated in the zeppelin-site.xml in order to enable HSTS. You can choose appropriate value for "max-age".```xml zeppelin.server.strict.transport max-age=631138519 The HTTP Strict-Transport-Security response header is a security feature that lets a web site tell browsers that it should only be communicated with using HTTPS, instead of using HTTP. Enable this when Zeppelin is running on HTTPS. Value is in Seconds, the default value is equivalent to 20 years.```Possible values are:* max-age=* max-age=; includeSubDomains* max-age=; preloadRead more about HSTS [here](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security).## Setting up X-XSS-PROTECTION HeaderThe HTTP X-XSS-Protection response header is a feature of Internet Explorer, Chrome and Safari Web browsers that initiates configured action when they detect reflected cross-site scripting (XSS) attacks. The following property needs to be updated in the zeppelin-site.xml in order to set X-XSS-PROTECTION header. ```xml zeppelin.server.xxss.protection 1; mode=block The HTTP X-XSS-Protection response header is a feature of Internet Explorer, Chrome and Safari that stops pages from loading when they detect reflected cross-site scripting (XSS) attacks. When value is set to 1 and a cross-site scripting attack is detected, the browser will sanitize the page (remove the unsafe parts).```You can choose appropriate value from below.* 0 (Disables XSS filtering)* 1 (Enables XSS filtering. If a cross-site scripting attack is detected, the browser will sanitize the page.)* 1; mode=block (Enables XSS filtering. The browser will prevent rendering of the page if an attack is detected.)Read more about HTTP X-XSS-Protection response header [here](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection).## Setting up X-Frame-Options HeaderThe X-Frame-Options HTTP response header can indicate browser to avoid clickjacking attacks, by ensuring that their content is not embedded into other sites in a ``,`` or ``.The following property needs to be updated in the zeppelin-site.xml in order to set X-Frame-Options header.```xml zeppelin.server.xframe.options SAMEORIGIN The X-Frame-Options HTTP response header can be used to indicate whether or not a browser should be allowed to render a page in a frame/iframe/object.```You can choose appropriate value from below.* `DENY`* `SAMEORIGIN`* `ALLOW-FROM uri`## Setting up Server HeaderSecurity conscious organisations does not want to reveal the Application Server name and version to prevent finding this information easily by Attacker while fingerprinting the Application. The exact version number can tell an Attacker if the current Application Server is patched for or vulnerable to certain publicly known CVE associated to it.The following property needs to be updated in the zeppelin-site.xml in order to set Server header.```xml zeppelin.server.jetty.name Jetty(7.6.0.v20120127) Hardcoding Application Server name to Prevent Fingerprinting```The value can be any "String".",
"url": " /setup/security/http_security_headers.html",
"group": "setup/security",
"excerpt": "There are multiple HTTP Security Headers which can be configured in Apache Zeppelin. This page describes how to enable them by providing appropriate value in Zeppelin configuration file."
}
,
"/setup/security/notebook_authorization.html": {
"title": "Notebook Authorization in Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Zeppelin Notebook Authorization## OverviewWe assume that there is an **Shiro Authentication** component that associates a user string and a set of group strings with every NotebookSocket.If you don't set the authentication components yet, please check [Shiro authentication for Apache Zeppelin](./shiro_authentication.html) first.## Authorization SettingYou can set Zeppelin notebook permissions in each notebooks. Of course only **notebook owners** can change this configuration.Just click **Lock icon** and open the permission setting page in your notebook.As you can see, each Zeppelin notebooks has 3 entities :* Owners ( users or groups )* Readers ( users or groups )* Writers ( users or groups )* Runners ( users or groups )Fill out the each forms with comma seperated **users** and **groups** configured in `conf/shiro.ini` file.If the form is empty (*), it means that any users can perform that operation.If someone who doesn't have **read** permission is trying to access the notebook or someone who doesn't have **write** permission is trying to edit the notebook,or someone who doesn't have **run** permission is trying to run a paragraph Zeppelin will ask to login or block the user.By default, owners and writers have **write** permission, owners, writers and runners have **run** permission, owners, writers, runners and readers have **read** permission## Separate notebook workspaces (public vs. private)By default, the authorization rights allow other users to see the newly created note, meaning the workspace is `public`. This behavior is controllable and can be set through either `ZEPPELIN_NOTEBOOK_PUBLIC` variable in `conf/zeppelin-env.sh`, or through `zeppelin.notebook.public` property in `conf/zeppelin-site.xml`. Thus, in order to make newly created note appear only in your `private` workspace by default, you can set either `ZEPPELIN_NOTEBOOK_PUBLIC` to `false` in your `conf/zeppelin-env.sh` as follows:```bashexport ZEPPELIN_NOTEBOOK_PUBLIC="false"```or set `zeppelin.notebook.public` property to `false` in `conf/zeppelin-site.xml` as follows:```xml zeppelin.notebook.public false Make notebook public by default when created, private otherwise```Behind the scenes, when you create a new note only the `owners` field is filled with current user, leaving `readers`, `runners` and `writers` fields empty. All the notes with at least one empty authorization field are considered to be in `public` workspace. Thus when setting `zeppelin.notebook.public` (or corresponding `ZEPPELIN_NOTEBOOK_PUBLIC`) to false, newly created notes have `readers`, `runners`, `writers` fields filled with current user, making note appear as in `private` workspace.## How it worksIn this section, we will explain the detail about how the notebook authorization works in backend side.### NotebookServerThe [NotebookServer](https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java) classifies every notebook operations into three categories: **Read**, **Run**, **Write**, **Manage**.Before executing a notebook operation, it checks if the user and the groups associated with the `NotebookSocket` have permissions.For example, before executing a **Read** operation, it checks if the user and the groups have at least one entity that belongs to the **Reader** entities.### Notebook REST API callZeppelin executes a [REST API call](https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java) for the notebook permission information.In the backend side, Zeppelin gets the user information for the connection and allows the operation if the users and groupsassociated with the current user have at least one entity that belongs to owner entities for the notebook.",
"url": " /setup/security/notebook_authorization.html",
"group": "setup/security",
"excerpt": "This page will guide you how you can set the permission for Zeppelin notebooks. This document assumes that Apache Shiro authentication was set up."
}
,
"/setup/security/datasource_authorization.html": {
"title": "Data Source Authorization in Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Data Source Authorization in Apache Zeppelin## OverviewData source authorization involves authenticating to the data source like a Mysql database and letting it determine user permissions.Apache Zeppelin allows users to use their own credentials to authenticate with **Data Sources**.For example, let's assume you have an account in the Vertica databases with credentials. You might want to use this account to create a JDBC connection instead of a shared account with all users who are defined in `conf/shiro.ini`. In this case, you can add your credential information to Apache Zeppelin and use them with below simple steps. ## How to save the credential information?You can add new credentials in the dropdown menu for your data source which can be passed to interpreters. **Entity** can be the key that distinguishes each credential sets.(We suggest that the convention of the **Entity** is `[Interpreter Group].[Interpreter Name]`.)Please see [what is interpreter group](../../usage/interpreter/overview.html#what-is-interpreter-group) for the detailed information.Type **Username & Password** for your own credentials. ex) Mysql user & password of the JDBC Interpreter.The credentials saved as per users defined in `conf/shiro.ini`.If you didn't activate [shiro authentication in Apache Zeppelin](./shiro_authentication.html), your credential information will be saved as `anonymous`.All credential information also can be found in `conf/credentials.json`. #### JDBC interpreterYou need to maintain per-user connection pools.The interpret method takes the user string as a parameter and executes the jdbc call using a connection in the user's connection pool.#### Presto You don't need a password if the Presto DB server runs backend code using HDFS authorization for the user.#### Vertica and Mysql You have to store the password information for users.## Please noteAs a first step of data source authentication feature, [ZEPPELIN-828](https://issues.apache.org/jira/browse/ZEPPELIN-828) was proposed and implemented in Pull Request [#860](https://github.com/apache/zeppelin/pull/860).Currently, only customized 3rd party interpreters can use this feature. We are planning to apply this mechanism to [the community managed interpreters](../../usage/interpreter/installation.html#available-community-managed-interpreters) in the near future. Please keep track [ZEPPELIN-1070](https://issues.apache.org/jira/browse/ZEPPELIN-1070). ",
"url": " /setup/security/datasource_authorization.html",
"group": "setup/security",
"excerpt": "Apache Zeppelin supports protected data sources. In case of a MySql database, every users can set up their own credentials to access it."
}
,
"/setup/security/shiro_authentication.html": {
"title": "Apache Shiro Authentication for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Apache Shiro authentication for Apache Zeppelin## Overview[Apache Shiro](http://shiro.apache.org/) is a powerful and easy-to-use Java security framework that performs authentication, authorization, cryptography, and session management. In this documentation, we will explain step by step how Shiro works for Zeppelin notebook authentication.When you connect to Apache Zeppelin, you will be asked to enter your credentials. Once you logged in, then you have access to all notes including other user's notes.## Security SetupYou can setup **Zeppelin notebook authentication** in some simple steps.### 1. Enable ShiroBy default in `conf`, you will find `shiro.ini.template`, this file is used as an example and it is strongly recommendedto create a `shiro.ini` file by doing the following command line```bashcp conf/shiro.ini.template conf/shiro.ini```For the further information about `shiro.ini` file format, please refer to [Shiro Configuration](http://shiro.apache.org/configuration.html#Configuration-INISections).### 2. Secure the Websocket channelSet to property **zeppelin.anonymous.allowed** to **false** in `conf/zeppelin-site.xml`. If you don't have this file yet, just copy `conf/zeppelin-site.xml.template` to `conf/zeppelin-site.xml`.### 3. Start Zeppelin```bashbin/zeppelin-daemon.sh start #(or restart)```Then you can browse Zeppelin at [http://localhost:8080](http://localhost:8080).### 4. LoginFinally, you can login using one of the below **username/password** combinations.```[users]admin = password1, adminuser1 = password2, role1, role2user2 = password3, role3user3 = password4, role2```You can set the roles for each users next to the password.## Groups and permissions (optional)In case you want to leverage user groups and permissions, use one of the following configuration for LDAP or AD under `[main]` segment in `shiro.ini`.```activeDirectoryRealm = org.apache.zeppelin.realm.ActiveDirectoryGroupRealmactiveDirectoryRealm.systemUsername = userNameAactiveDirectoryRealm.systemPassword = passwordAactiveDirectoryRealm.searchBase = CN=Users,DC=SOME_GROUP,DC=COMPANY,DC=COMactiveDirectoryRealm.url = ldap://ldap.test.com:389activeDirectoryRealm.groupRolesMap = "CN=aGroupName,OU=groups,DC=SOME_GROUP,DC=COMPANY,DC=COM":"group1"activeDirectoryRealm.authorizationCachingEnabled = falseactiveDirectoryRealm.principalSuffix = @corp.company.netldapRealm = org.apache.zeppelin.realm.LdapGroupRealm# search base for ldap groups (only relevant for LdapGroupRealm):ldapRealm.contextFactory.environment[ldap.searchBase] = dc=COMPANY,dc=COMldapRealm.contextFactory.url = ldap://ldap.test.com:389ldapRealm.userDnTemplate = uid={0},ou=Users,dc=COMPANY,dc=COMldapRealm.contextFactory.authenticationMechanism = simple```also define roles/groups that you want to have in system, like below;```[roles]admin = *hr = *finance = *group1 = *```## Configure Realm (optional)Realms are responsible for authentication and authorization in Apache Zeppelin. By default, Apache Zeppelin uses [IniRealm](https://shiro.apache.org/static/latest/apidocs/org/apache/shiro/realm/text/IniRealm.html) (users and groups are configurable in `conf/shiro.ini` file under `[user]` and `[group]` section). You can also leverage Shiro Realms like [JndiLdapRealm](https://shiro.apache.org/static/latest/apidocs/org/apache/shiro/realm/ldap/JndiLdapRealm.html), [JdbcRealm](https://shiro.apache.org/static/latest/apidocs/org/apache/shiro/realm/jdbc/JdbcRealm.html) or create [our own](https://shiro.apache.org/static/latest/apidocs/org/apache/shiro/realm/AuthorizingRealm.html).To learn more about Apache Shiro Realm, please check [this documentation](http://shiro.apache.org/realm.html).We also provide community custom Realms.**Note**: When using any of the below realms the default password-based (IniRealm) authentication needs to be disabled.### Active Directory```activeDirectoryRealm = org.apache.zeppelin.realm.ActiveDirectoryGroupRealmactiveDirectoryRealm.systemUsername = userNameAactiveDirectoryRealm.systemPassword = passwordAactiveDirectoryRealm.hadoopSecurityCredentialPath = jceks://file/user/zeppelin/conf/zeppelin.jceksactiveDirectoryRealm.searchBase = CN=Users,DC=SOME_GROUP,DC=COMPANY,DC=COMactiveDirectoryRealm.url = ldap://ldap.test.com:389activeDirectoryRealm.groupRolesMap = "CN=aGroupName,OU=groups,DC=SOME_GROUP,DC=COMPANY,DC=COM":"group1"activeDirectoryRealm.authorizationCachingEnabled = falseactiveDirectoryRealm.principalSuffix = @corp.company.net```Also instead of specifying systemPassword in clear text in shiro.ini administrator can choose to specify the same in "hadoop credential".Create a keystore file using the hadoop credential commandline, for this the hadoop commons should be in the classpath`hadoop credential create activeDirectoryRealm.systempassword -provider jceks://file/user/zeppelin/conf/zeppelin.jceks`Change the following values in the Shiro.ini file, and uncomment the line:`activeDirectoryRealm.hadoopSecurityCredentialPath = jceks://file/user/zeppelin/conf/zeppelin.jceks`### LDAPTwo options exist for configuring an LDAP Realm. The simpler to use is the LdapGroupRealm. How ever it has limitedflexibility with mapping of ldap groups to users and for authorization for user groups. A sample configuration file forthis realm is given below.```ldapRealm = org.apache.zeppelin.realm.LdapGroupRealm# search base for ldap groups (only relevant for LdapGroupRealm):ldapRealm.contextFactory.environment[ldap.searchBase] = dc=COMPANY,dc=COMldapRealm.contextFactory.url = ldap://ldap.test.com:389ldapRealm.userDnTemplate = uid={0},ou=Users,dc=COMPANY,dc=COMldapRealm.contextFactory.authenticationMechanism = simple```The other more flexible option is to use the LdapRealm. It allows for mapping of ldapgroups to roles and also allows for role/group based authentication into the zeppelin server. Sample configuration for this realm is given below.```[main]ldapRealm=org.apache.zeppelin.realm.LdapRealmldapRealm.contextFactory.authenticationMechanism=simpleldapRealm.contextFactory.url=ldap://localhost:33389ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org# Ability to set ldap paging Size if needed default is 100ldapRealm.pagingSize = 200ldapRealm.authorizationEnabled=trueldapRealm.contextFactory.systemAuthenticationMechanism=simpleldapRealm.searchBase=dc=hadoop,dc=apache,dc=orgldapRealm.userSearchBase = dc=hadoop,dc=apache,dc=orgldapRealm.groupSearchBase = ou=groups,dc=hadoop,dc=apache,dc=orgldapRealm.groupObjectClass=groupofnames# Allow userSearchAttribute to be customizedldapRealm.userSearchAttributeName = sAMAccountNameldapRealm.memberAttribute=member# force usernames returned from ldap to lowercase useful for ADldapRealm.userLowerCase = true# ability set searchScopes subtree (default), one, baseldapRealm.userSearchScope = subtree;ldapRealm.groupSearchScope = subtree;ldapRealm.memberAttributeValueTemplate=cn={0},ou=people,dc=hadoop,dc=apache,dc=orgldapRealm.contextFactory.systemUsername=uid=guest,ou=people,dc=hadoop,dc=apache,dc=orgldapRealm.contextFactory.systemPassword=S{ALIAS=ldcSystemPassword}# enable support for nested groups using the LDAP_MATCHING_RULE_IN_CHAIN operatorldapRealm.groupSearchEnableMatchingRuleInChain = true# optional mapping from physical groups to logical application rolesldapRealm.rolesByGroup = LDN_USERS: user_role, NYK_USERS: user_role, HKG_USERS: user_role, GLOBAL_ADMIN: admin_role# optional list of roles that are allowed to authenticate. Incase not present all groups are allowed to authenticate (login).# This changes nothing for url specific permissions that will continue to work as specified in [urls].ldapRealm.allowedRolesForAuthentication = admin_role,user_roleldapRealm.permissionsByRole= user_role = *:ToDoItemsJdo:*:*, *:ToDoItem:*:*; admin_role = *securityManager.sessionManager = $sessionManagersecurityManager.realms = $ldapRealm```Also instead of specifying systemPassword in clear text in `shiro.ini` administrator can choose to specify the same in "hadoop credential". Create a keystore file using the hadoop credential command line:``` hadoop credential create ldapRealm.systemPassword -provider jceks://file/user/zeppelin/conf/zeppelin.jceks```Add the following line in the `shiro.ini` file:``` ldapRealm.hadoopSecurityCredentialPath = jceks://file/user/zeppelin/conf/zeppelin.jceks```### PAM[PAM](https://en.wikipedia.org/wiki/Pluggable_authentication_module) authentication support allows the reuse of existing authenticationmoduls on the host where Zeppelin is running. On a typical system modules are configured per service for example sshd, passwd, etc. under `/etc/pam.d/`. You caneither reuse one of these services or create your own for Zeppelin. Activiting PAM authentication requires two parameters: 1. realm: The Shiro realm being used 2. service: The service configured under `/etc/pam.d/` to be used. The name here needs to be the same as the file name under `/etc/pam.d/````[main] pamRealm=org.apache.zeppelin.realm.PamRealm pamRealm.service=sshd```### ZeppelinHub[ZeppelinHub](https://www.zeppelinhub.com) is a service that synchronize your Apache Zeppelin notebooks and enables you to collaborate easily.To enable login with your ZeppelinHub credential, apply the following change in `conf/shiro.ini` under `[main]` section.```### A sample for configuring ZeppelinHub RealmzeppelinHubRealm = org.apache.zeppelin.realm.ZeppelinHubRealm## Url of ZeppelinHubzeppelinHubRealm.zeppelinhubUrl = https://www.zeppelinhub.comsecurityManager.realms = $zeppelinHubRealm```> Note: ZeppelinHub is not releated to Apache Zeppelin project.### Knox SSO[KnoxSSO](https://knox.apache.org/books/knox-0-13-0/dev-guide.html#KnoxSSO+Integration) provides an abstraction for integrating any number of authentication systems and SSO solutions and enables participating web applications to scale to those solutions more easily. Without the token exchange capabilities offered by KnoxSSO each component UI would need to integrate with each desired solution on its own.To enable this, apply the following change in `conf/shiro.ini` under `[main]` section.```### A sample for configuring Knox JWT RealmknoxJwtRealm = org.apache.zeppelin.realm.jwt.KnoxJwtRealm## Domain of Knox SSOknoxJwtRealm.providerUrl = https://domain.example.com/## Url for loginknoxJwtRealm.login = gateway/knoxsso/knoxauth/login.html## Url for logoutknoxJwtRealm.logout = gateway/knoxssout/api/v1/webssoutknoxJwtRealm.redirectParam = originalUrlknoxJwtRealm.cookieName = hadoop-jwtknoxJwtRealm.publicKeyPath = /etc/zeppelin/conf/knox-sso.pemknoxJwtRealm.groupPrincipalMapping = group.principal.mappingknoxJwtRealm.principalMapping = principal.mapping# This is required if KNOX SSO is enabled, to check if "knoxJwtRealm.cookieName" cookie was expired/deleted. authc = org.apache.zeppelin.realm.jwt.KnoxAuthenticationFilter```## Secure Cookie for Zeppelin Sessions (optional)Zeppelin can be configured to set `HttpOnly` flag in the session cookie. With this configuration, Zeppelin cookies can not be accessed via client side scripts thus preventing majority of Cross-site scripting (XSS) attacks.To enable secure cookie support via Shiro, add the following lines in `conf/shiro.ini` under `[main]` section, afterdefining a `sessionManager`.```cookie = org.apache.shiro.web.servlet.SimpleCookiecookie.name = JSESSIONIDcookie.secure = truecookie.httpOnly = truesessionManager.sessionIdCookie = $cookie```## Secure your Zeppelin information (optional)By default, anyone who defined in `[users]` can share **Interpreter Setting**, **Credential** and **Configuration** information in Apache Zeppelin.Sometimes you might want to hide these information for your use case.Since Shiro provides **url-based security**, you can hide the information by commenting or uncommenting these below lines in `conf/shiro.ini`.```[urls]/api/interpreter/** = authc, roles[admin]/api/configurations/** = authc, roles[admin]/api/credential/** = authc, roles[admin]```In this case, only who have `admin` role can see **Interpreter Setting**, **Credential** and **Configuration** information.If you want to grant this permission to other users, you can change **roles[ ]** as you defined at `[users]` section.### Apply multiple roles in Shiro configurationBy default, Shiro will allow access to a URL if only user is part of "**all the roles**" defined like this:```[urls]/api/interpreter/** = authc, roles[admin, role1]```### Apply multiple roles or user in Shiro configurationIf there is a need that user with "**any of the defined roles or user itself**" should be allowed, then following Shiro configuration can be used:```[main]anyofrolesuser = org.apache.zeppelin.utils.AnyOfRolesUserAuthorizationFilter[urls]/api/interpreter/** = authc, anyofrolesuser[admin, user1]/api/configurations/** = authc, roles[admin]/api/credential/** = authc, roles[admin]```> **NOTE :** All of the above configurations are defined in the `conf/shiro.ini` file.## Other authentication methods- [HTTP Basic Authentication using NGINX](./authentication_nginx.html)",
"url": " /setup/security/shiro_authentication.html",
"group": "setup/security",
"excerpt": "Apache Shiro is a powerful and easy-to-use Java security framework that performs authentication, authorization, cryptography, and session management. This document explains step by step how Shiro can be used for Zeppelin notebook authentication."
}
,
"/setup/security/authentication_nginx.html": {
"title": "HTTP Basic Auth using NGINX",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Authentication for NGINX[Build in authentication mechanism](./shiro_authentication.html) is recommended way for authentication. In case of you want authenticate using NGINX and [HTTP basic auth](https://en.wikipedia.org/wiki/Basic_access_authentication), please read this document.## HTTP Basic Authentication using NGINX> **Quote from Wikipedia:** NGINX is a web server. It can act as a reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer and an HTTP cache.So you can use NGINX server as proxy server to serve HTTP Basic Authentication as a separate process along with Zeppelin server.Here are instructions how to accomplish the setup NGINX as a front-end authentication server and connect Zeppelin at behind.This instruction based on Ubuntu 14.04 LTS but may work with other OS with few configuration changes.1. Install NGINX server on your server instance You can install NGINX server with same box where zeppelin installed or separate box where it is dedicated to serve as proxy server. ```bash $ apt-get install nginx ``` > **NOTE :** On pre 1.3.13 version of NGINX, Proxy for Websocket may not fully works. Please use latest version of NGINX. See: [NGINX documentation](https://www.nginx.com/blog/websocket-nginx/).1. Setup init script in NGINX In most cases, NGINX configuration located under `/etc/nginx/sites-available`. Create your own configuration or add your existing configuration at `/etc/nginx/sites-available`. ```bash $ cd /etc/nginx/sites-available $ touch my-zeppelin-auth-setting ``` Now add this script into `my-zeppelin-auth-setting` file. You can comment out `optional` lines If you want serve Zeppelin under regular HTTP 80 Port. ``` upstream zeppelin { server [YOUR-ZEPPELIN-SERVER-IP]:[YOUR-ZEPPELIN-SERVER-PORT]; # For security, It is highly recommended to make this address/port as non-public accessible } # Zeppelin Website server { listen [YOUR-ZEPPELIN-WEB-SERVER-PORT]; listen 443 ssl; # optional, to serve HTTPS connection server_name [YOUR-ZEPPELIN-SERVER-HOST]; # for example: zeppelin.mycompany.com ssl_certificate [PATH-TO-YOUR-CERT-FILE]; # optional, to serve HTTPS connection ssl_certificate_key [PATH-TO-YOUR-CERT-KEY-FILE]; # optional, to serve HTTPS connection if ($ssl_protocol = "") { rewrite ^ https://$host$request_uri? permanent; # optional, to force use of HTTPS } location / { # For regular websever support proxy_pass http://zeppelin; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_set_header X-NginX-Proxy true; proxy_redirect off; auth_basic "Restricted"; auth_basic_user_file /etc/nginx/.htpasswd; } location /ws { # For websocket support proxy_pass http://zeppelin/ws; proxy_http_version 1.1; proxy_set_header Upgrade websocket; proxy_set_header Connection upgrade; proxy_read_timeout 86400; } } ``` Then make a symbolic link to this file from `/etc/nginx/sites-enabled/` to enable configuration above when NGINX reloads. ```bash $ ln -s /etc/nginx/sites-enabled/my-zeppelin-auth-setting /etc/nginx/sites-available/my-zeppelin-auth-setting ```1. Setup user credential into `.htpasswd` file and restart server Now you need to setup `.htpasswd` file to serve list of authenticated user credentials for NGINX server. ```bash $ cd /etc/nginx $ htpasswd -c htpasswd [YOUR-ID] NEW passwd: [YOUR-PASSWORD] RE-type new passwd: [YOUR-PASSWORD-AGAIN] ``` Or you can use your own apache `.htpasswd` files in other location for setting up property: `auth_basic_user_file` Restart NGINX server. ```bash $ service nginx restart ``` Then check HTTP Basic Authentication works in browser. If you can see regular basic auth popup and then able to login with credential you entered into `.htpasswd` you are good to go.1. More security consideration* Using HTTPS connection with Basic Authentication is highly recommended since basic auth without encryption may expose your important credential information over the network.* Using [Shiro Security feature built-into Zeppelin](./shiro_authentication.html) is recommended if you prefer all-in-one solution for authentication but NGINX may provides ad-hoc solution for re-use authentication served by your system's NGINX server or in case of you need to separate authentication from zeppelin server.* It is recommended to isolate direct connection to Zeppelin server from public internet or external services to secure your zeppelin instance from unexpected attack or problems caused by public zone.## Another optionAnother option is to have an authentication server that can verify user credentials in an LDAP server.If an incoming request to the Zeppelin server does not have a cookie with user information encrypted with the authentication server public key, the useris redirected to the authentication server. Once the user is verified, the authentication server redirects the browser to a specific URL in the Zeppelin server which sets the authentication cookie in the browser.The end result is that all requests to the Zeppelin web server have the authentication cookie which contains user and groups information.",
"url": " /setup/security/authentication_nginx.html",
"group": "setup/security",
"excerpt": "There are multiple ways to enable authentication in Apache Zeppelin. This page describes HTTP basic auth using NGINX."
}
,
"/setup/storage/storage.html": {
"title": "Notebook Storage for Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Notebook storage options for Apache Zeppelin## OverviewApache Zeppelin has a pluggable notebook storage mechanism controlled by `zeppelin.notebook.storage` configuration option with multiple implementations.There are few notebook storage systems available for a use out of the box: * (default) use local file system and version it using local Git repository - `GitNotebookRepo` * all notes are saved in the notebook folder in your local File System - `VFSNotebookRepo` * all notes are saved in the notebook folder in hadoop compatible file system - `FileSystemNotebookRepo` * storage using Amazon S3 service - `S3NotebookRepo` * storage using Azure service - `AzureNotebookRepo` * storage using Google Cloud Storage - `GCSNotebookRepo` * storage using MongoDB - `MongoNotebookRepo` * storage using GitHub - `GitHubNotebookRepo`Multiple storage systems can be used at the same time by providing a comma-separated list of the class-names in the configuration.By default, only first two of them will be automatically kept in sync by Zeppelin.## Notebook Storage in local Git repository To enable versioning for all your local notebooks though a standard Git repository - uncomment the next property in `zeppelin-site.xml` in order to use GitNotebookRepo class:```xml zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.GitNotebookRepo notebook persistence layer implementation```## Notebook Storage in hadoop compatible file system repository Notes may be stored in hadoop compatible file system such as hdfs, so that multiple Zeppelin instances can share the same notes. It supports all the versions of hadoop 2.x. If you use `FileSystemNotebookRepo`, then `zeppelin.notebook.dir` is the path on the hadoop compatible file system. And you need to specify `HADOOP_CONF_DIR` in `zeppelin-env.sh` so that zeppelin can find the right hadoop configuration files.If your hadoop cluster is kerberized, then you need to specify `zeppelin.server.kerberos.keytab` and `zeppelin.server.kerberos.principal````xml zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.FileSystemNotebookRepo hadoop compatible file system notebook persistence layer implementation```## Notebook Storage in S3 Notebooks may be stored in S3, and optionally encrypted. The [``DefaultAWSCredentialsProviderChain``](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html) credentials provider is used for credentials and checks the following:- The ``AWS_ACCESS_KEY_ID`` and ``AWS_SECRET_ACCESS_KEY`` environment variables- The ``aws.accessKeyId`` and ``aws.secretKey`` Java System properties- Credential profiles file at the default location (````~/.aws/credentials````) used by the AWS CLI- Instance profile credentials delivered through the Amazon EC2 metadata serviceThe following folder structure will be created in S3:```s3://bucket_name/username/notebook-id/```Configure by setting environment variables in the file **zeppelin-env.sh**:```bashexport ZEPPELIN_NOTEBOOK_S3_BUCKET=bucket_nameexport ZEPPELIN_NOTEBOOK_S3_USER=username```Or using the file **zeppelin-site.xml** uncomment and complete the S3 settings:```xml zeppelin.notebook.s3.bucket bucket_name bucket name for notebook storage zeppelin.notebook.s3.user username user name for s3 folder structure```Uncomment the next property for use S3NotebookRepo class:```xml zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.S3NotebookRepo notebook persistence layer implementation```Comment out the next property to disable local git notebook storage (the default):```xml zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.GitNotebookRepo versioned notebook persistence layer implementation```### Data Encryption in S3#### AWS KMS encryption keysTo use an [AWS KMS](https://aws.amazon.com/kms/) encryption key to encrypt notebooks, set the following environment variable in the file **zeppelin-env.sh**:```bashexport ZEPPELIN_NOTEBOOK_S3_KMS_KEY_ID=kms-key-id```Or using the following setting in **zeppelin-site.xml**:```xml zeppelin.notebook.s3.kmsKeyID AWS-KMS-Key-UUID AWS KMS key ID used to encrypt notebook data in S3```In order to set custom KMS key region, set the following environment variable in the file **zeppelin-env.sh**:```bashexport ZEPPELIN_NOTEBOOK_S3_KMS_KEY_REGION=kms-key-region```Or using the following setting in **zeppelin-site.xml**:```xml zeppelin.notebook.s3.kmsKeyRegion target-region AWS KMS key region in your AWS account```Format of `target-region` is described in more details [here](http://docs.aws.amazon.com/general/latest/gr/rande.html#kms_region) in second `Region` column (e.g. `us-east-1`).#### Custom Encryption Materials Provider classYou may use a custom [``EncryptionMaterialsProvider``](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/EncryptionMaterialsProvider.html) class as long as it is available in the classpath and able to initialize itself from system properties or another mechanism. To use this, set the following environment variable in the file **zeppelin-env.sh**:```bashexport ZEPPELIN_NOTEBOOK_S3_EMP=class-name```Or using the following setting in **zeppelin-site.xml**:```xml zeppelin.notebook.s3.encryptionMaterialsProvider provider implementation class name Custom encryption materials provider used to encrypt notebook data in S3``` #### Enable server-side encryptionTo request server-side encryption of notebooks, set the following environment variable in the file **zeppelin-env.sh**:```bashexport ZEPPELIN_NOTEBOOK_S3_SSE=true```Or using the following setting in **zeppelin-site.xml**:```xml zeppelin.notebook.s3.sse true Server-side encryption enabled for notebooks```## Notebook Storage in Azure Using `AzureNotebookRepo` you can connect your Zeppelin with your Azure account for notebook storage.First of all, input your `AccountName`, `AccountKey`, and `Share Name` in the file **zeppelin-site.xml** by commenting out and completing the next properties:```xml zeppelin.notebook.azure.connectionString DefaultEndpointsProtocol=https;AccountName=;AccountKey= Azure account credentials zeppelin.notebook.azure.share zeppelin share name for notebook storage```Secondly, you can initialize `AzureNotebookRepo` class in the file **zeppelin-site.xml** by commenting the next property:```xml zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.GitNotebookRepo versioned notebook persistence layer implementation```and commenting out:```xml zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.AzureNotebookRepo notebook persistence layer implementation```In case you want to use simultaneously your local git storage with Azure storage use the following property instead: ```xml zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.GitNotebookRepo, apache.zeppelin.notebook.repo.AzureNotebookRepo notebook persistence layer implementation```Optionally, you can specify Azure folder structure name in the file **zeppelin-site.xml** by commenting out the next property: ```xml zeppelin.notebook.azure.user user optional user name for Azure folder structure```## Notebook Storage in Google Cloud StorageUsing `GCSNotebookRepo` you can connect Zeppelin with Google Cloud Storage using [Application Default Credentials](https://cloud.google.com/docs/authentication/production).First, choose a GCS path under which to store notebooks.```xml zeppelin.notebook.gcs.dir A GCS path in the form gs://bucketname/path/to/dir. Notes are stored at {zeppelin.notebook.gcs.dir}/{notebook-id}/note.json ```Then, initialize the `GCSNotebookRepo` class in the file **zeppelin-site.xml** by commenting the next property:```xml zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.GitNotebookRepo versioned notebook persistence layer implementation```and commenting out:```xml zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.GCSNotebookRepo notebook persistence layer implementation```Or, if you want to simultaneously use your local git storage with GCS, use the following property instead:```xml zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.GitNotebookRepo,org.apache.zeppelin.notebook.repo.GCSNotebookRepo notebook persistence layer implementation```### Google Cloud API AuthenticationNote: On Google App Engine, Google Cloud Shell, and Google Compute Engine, thesesteps are not necessary, as build-in credentials are used by default.For more information, see [Application Default Credentials](https://cloud.google.com/docs/authentication/production)#### Using gcloud auth application-default loginSee the [gcloud docs](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login)As the user running the zeppelin daemon, run:```bashgcloud auth application-default login```You can also use `--scopes` to restrict access to specific Google APIs, such asCloud Storage and BigQuery.#### Using service account key filesAlternatively, to use a [service account](https://cloud.google.com/compute/docs/access/service-accounts)for authentication with GCS, you will need a JSON service account key file.1. Navigate to the [service accounts page](https://console.cloud.google.com/iam-admin/serviceaccounts/project)2. Click `CREATE SERVICE ACCOUNT`3. Select at least `Storage -> Storage Object Admin`. Note that this is **different** than `Storage Admin`.4. If you are also using the BigQuery Interpreter, add the appropriate permissions (e.g. `Bigquery -> Bigquery Data Viewer and BigQuery User`)5. Name your service account, and select "Furnish a new private key" to download a `.json` file. Click "Create".6. Move the downloaded file to a location of your choice (e.g. `/path/to/my/key.json`), and give it appropriate permissions. Ensure at least the user running the zeppelin daemon can read it.Then, point `GOOGLE_APPLICATION_CREDENTIALS` at your new key file in **zeppelin-env.sh**. For example:```bashexport GOOGLE_APPLICATION_CREDENTIALS=/path/to/my/key.json```## Notebook Storage in ZeppelinHub ZeppelinHub storage layer allows out of the box connection of Zeppelin instance with your ZeppelinHub account. First of all, you need to either comment out the following property in **zeppelin-site.xml**:```xml<!-- zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.GitNotebookRepo, org.apache.zeppelin.notebook.repo.zeppelinhub.ZeppelinHubRepo two notebook persistence layers (local + ZeppelinHub)-->```or set the environment variable in the file **zeppelin-env.sh**:```bashexport ZEPPELIN_NOTEBOOK_STORAGE="org.apache.zeppelin.notebook.repo.GitNotebookRepo, org.apache.zeppelin.notebook.repo.zeppelinhub.ZeppelinHubRepo"```Secondly, you need to set the environment variables in the file **zeppelin-env.sh**:```bashexport ZEPPELINHUB_API_TOKEN=ZeppelinHub tokenexport ZEPPELINHUB_API_ADDRESS=address of ZeppelinHub service (e.g. https://www.zeppelinhub.com)```You can get more information on generating `token` and using authentication on the corresponding [help page](http://help.zeppelinhub.com/zeppelin_integration/#add-a-new-zeppelin-instance-and-generate-a-token).## Notebook Storage in MongoDB Using `MongoNotebookRepo`, you can store your notebook in [MongoDB](https://www.mongodb.com/).### Why MongoDB?* **[High Availability (HA)](https://en.wikipedia.org/wiki/High_availability)** by a [replica set](https://docs.mongodb.com/manual/reference/glossary/#term-replica-set)* Seperation of storage from server### How to useYou can use MongoDB as notebook storage by editting `zeppelin-env.sh` or `zeppelin-site.xml`.#### (Method 1) by editting `zeppelin-env.sh`Add a line below to `$ZEPPELIN_HOME/conf/zeppelin-env.sh`:```bashexport ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.MongoNotebookRepo```> *NOTE:* The default MongoDB connection URI is `mongodb://localhost`#### (Method 2) by editting `zeppelin-site.xml`Or, **uncomment** lines below at `$ZEPPELIN_HOME/conf/zeppelin-site.xml`:```xml zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.MongoNotebookRepo notebook persistence layer implementation```And **comment** lines below:```xml zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.GitNotebookRepo versioned notebook persistence layer implementation```### Configurable OptionsYou can configure options below in `zeppelin-env.sh`.* `ZEPPELIN_NOTEBOOK_MONGO_URI` [MongoDB connection URI](https://docs.mongodb.com/manual/reference/connection-string/) used to connect to a MongoDB database server* `ZEPPELIN_NOTEBOOK_MONGO_DATABASE` Database name* `ZEPPELIN_NOTEBOOK_MONGO_COLLECTION` Collection name* `ZEPPELIN_NOTEBOOK_MONGO_AUTOIMPORT` If `true`, import local notes (refer to description below for details)Or, you can configure them in `zeppelin-site.xml`. Corresponding option names as follows:* `zeppelin.notebook.mongo.uri`* `zeppelin.notebook.mongo.database`* `zeppelin.notebook.mongo.collection`* `zeppelin.notebook.mongo.autoimport`#### Example configurations in `zeppelin-env.sh````shexport ZEPPELIN_NOTEBOOK_MONGO_URI=mongodb://db1.example.com:27017export ZEPPELIN_NOTEBOOK_MONGO_DATABASE=myfancyexport ZEPPELIN_NOTEBOOK_MONGO_COLLECTION=notebookexport ZEPPELIN_NOTEBOOK_MONGO_AUTOIMPORT=true```#### Import your local notes automaticallyBy setting `ZEPPELIN_NOTEBOOK_MONGO_AUTOIMPORT` as `true` (default `false`), you can import your local notes automatically when Zeppelin daemon starts up. This feature is for easy migration from local file system storage to MongoDB storage. A note with ID already existing in the collection will not be imported.## Notebook Storage in GitHubTo enable GitHub tracking, uncomment the following properties in `zeppelin-site.xml````xml zeppelin.notebook.git.remote.url remote Git repository URL zeppelin.notebook.git.remote.username token remote Git repository username zeppelin.notebook.git.remote.access-token remote Git repository password zeppelin.notebook.git.remote.origin origin Git repository remote```And set the `zeppelin.notebook.storage` propery to `org.apache.zeppelin.notebook.repo.GitHubNotebookRepo````xml zeppelin.notebook.storage org.apache.zeppelin.notebook.repo.GitHubNotebookRepo```The access token could be obtained by following the steps on this link https://github.com/settings/tokens.",
"url": " /setup/storage/storage.html",
"group": "setup/storage",
"excerpt": "Apache Zeppelin has a pluggable notebook storage mechanism controlled by zeppelin.notebook.storage configuration option with multiple implementations.\""
}
,
"/setup/deployment/virtual_machine.html": {
"title": "Apache Zeppelin on Vagrant Virtual Machine",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Apache Zeppelin on Vagrant Virtual Machine## OverviewApache Zeppelin distribution includes a script directory `scripts/vagrant/zeppelin-dev`This script creates a virtual machine that launches a repeatable, known set of core dependencies required for developing Zeppelin. It can also be used to run an existing Zeppelin build if you don't plan to build from source.For PySpark users, this script includes several helpful [Python Libraries](#python-extras).For SparkR users, this script includes several helpful [R Libraries](#r-extras).### PrerequisitesThis script requires three applications, [Ansible](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip "Ansible"), [Vagrant](http://www.vagrantup.com "Vagrant") and [Virtual Box](https://www.virtualbox.org/ "Virtual Box"). All of these applications are freely available as Open Source projects and extremely easy to set up on most operating systems.## Create a Zeppelin Ready VMIf you are running Windows and don't yet have python installed, [install Python 2.7.x](https://www.python.org/downloads/release/python-2710/) first.1. Download and Install Vagrant: [Vagrant Downloads](http://www.vagrantup.com/downloads.html)2. Install Ansible: [Ansible Python pip install](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip) ```bash sudo easy_install pip sudo pip install ansible ansible --version ``` After then, please check whether it reports **ansible version 1.9.2 or higher**.3. Install Virtual Box: [Virtual Box Downloads](https://www.virtualbox.org/ "Virtual Box")4. Type `vagrant up` from within the `/scripts/vagrant/zeppelin-dev` directoryThats it ! You can now run `vagrant ssh` and this will place you into the guest machines terminal prompt.If you don't wish to build Zeppelin from scratch, run the z-manager installer script while running in the guest VM:```bashcurl -fsSL https://raw.githubusercontent.com/NFLabs/z-manager/master/zeppelin-installer.sh | bash```## Building ZeppelinYou can now ```bashgit clone git://git.apache.org/zeppelin.git```into a directory on your host machine, or directly in your virtual machine.Cloning Zeppelin into the `/scripts/vagrant/zeppelin-dev` directory from the host, will allow the directory to be shared between your host and the guest machine.Cloning the project again may seem counter intuitive, since this script likely originated from the project repository. Consider copying just the vagrant/zeppelin-dev script from the Zeppelin project as a stand alone directory, then once again clone the specific branch you wish to build.Synced folders enable Vagrant to sync a folder on the host machine to the guest machine, allowing you to continue working on your project's files on your host machine, but use the resources in the guest machine to compile or run your project. _[(1) Synced Folder Description from Vagrant Up](https://docs.vagrantup.com/v2/synced-folders/index.html)_By default, Vagrant will share your project directory (the directory with the Vagrantfile) to `/vagrant`. Which means you should be able to build within the guest machine after you`cd /vagrant/zeppelin`## What's in this VM?Running the following commands in the guest machine should display these expected versions:* `node --version` should report *v0.12.7** `mvn --version` should report *Apache Maven 3.3.9* and *Java version: 1.7.0_85*The virtual machine consists of: - Ubuntu Server 14.04 LTS - Node.js 0.12.7 - npm 2.11.3 - ruby 1.9.3 + rake, make and bundler (only required if building jekyll documentation) - Maven 3.3.9 - Git - Unzip - libfontconfig to avoid phatomJs missing dependency issues - openjdk-7-jdk - Python addons: pip, matplotlib, scipy, numpy, pandas - [R](https://www.r-project.org/) and R Packages required to run the R Interpreter and the related R tutorial notebook, including: Knitr, devtools, repr, rCharts, ggplot2, googleVis, mplot, htmltools, base64enc, data.table## How to build & run ZeppelinThis assumes you've already cloned the project either on the host machine in the zeppelin-dev directory (to be shared with the guest machine) or cloned directly into a directory while running inside the guest machine. The following build steps will also include Python and R support via PySpark and SparkR:```bashcd /zeppelinmvn clean package -Pspark-1.6 -Phadoop-2.4 -DskipTests./bin/zeppelin-daemon.sh start```On your host machine browse to `http://localhost:8080/`If you [turned off port forwarding](#tweaking-the-virtual-machine) in the `Vagrantfile` browse to `http://192.168.51.52:8080`## Tweaking the Virtual MachineIf you plan to run this virtual machine along side other Vagrant images, you may wish to bind the virtual machine to a specific IP address, and not use port fowarding from your local host.Comment out the `forward_port` line, and uncomment the `private_network` line in Vagrantfile. The subnet that works best for your local network will vary so adjust `192.168.*.*` accordingly.```#config.vm.network "forwarded_port", guest: 8080, host: 8080config.vm.network "private_network", ip: "192.168.51.52"````vagrant halt` followed by `vagrant up` will restart the guest machine bound to the IP address of `192.168.51.52`.This approach usually is typically required if running other virtual machines that discover each other directly by IP address, such as Spark Masters and Slaves as well as Cassandra Nodes, Elasticsearch Nodes, and other Spark data sources. You may wish to launch nodes in virtual machines with IP addresses in a subnet that works for your local network, such as: 192.168.51.53, 192.168.51.54, 192.168.51.53, etc..## Extras### Python ExtrasWith Zeppelin running, **Numpy**, **SciPy**, **Pandas** and **Matplotlib** will be available. Create a pyspark notebook, and try the below code.```python%pysparkimport numpyimport scipyimport pandasimport matplotlibprint "numpy " + numpy.__version__print "scipy " + scipy.__version__print "pandas " + pandas.__version__print "matplotlib " + matplotlib.__version__```To Test plotting using Matplotlib into a rendered `%html` SVG image, try```python%pysparkimport matplotlibmatplotlib.use('Agg') # turn off interactive charting so this works for server side SVG renderingimport matplotlib.pyplot as pltimport numpy as npimport StringIO# clear out any previous plots on this noteplt.clf()def show(p): img = StringIO.StringIO() p.savefig(img, format='svg') img.seek(0) print "%html " + img.buf + ""# Example datapeople = ('Tom', 'Dick', 'Harry', 'Slim', 'Jim')y_pos = np.arange(len(people))performance = 3 + 10 * np.random.rand(len(people))error = np.random.rand(len(people))plt.barh(y_pos, performance, xerr=error, align='center', alpha=0.4)plt.yticks(y_pos, people)plt.xlabel('Performance')plt.title('How fast do you want to go today?')show(plt)```### R ExtrasWith zeppelin running, an R Tutorial notebook will be available. The R packages required to run the examples and graphs in this tutorial notebook were installed by this virtual machine.The installed R Packages include: `knitr`, `devtools`, `repr`, `rCharts`, `ggplot2`, `googleVis`, `mplot`, `htmltools`, `base64enc`, `data.table`.",
"url": " /setup/deployment/virtual_machine.html",
"group": "setup/deployment",
"excerpt": "Apache Zeppelin provides a script for running a virtual machine for development through Vagrant. The script will create a virtual machine with core dependencies pre-installed, required for developing Apache Zeppelin."
}
,
"/setup/deployment/spark_cluster_mode.html": {
"title": "Apache Zeppelin on Spark cluster mode",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Apache Zeppelin on Spark Cluster Mode## Overview[Apache Spark](http://spark.apache.org/) has supported three cluster manager types([Standalone](http://spark.apache.org/docs/latest/spark-standalone.html), [Apache Mesos](http://spark.apache.org/docs/latest/running-on-mesos.html) and [Hadoop YARN](http://spark.apache.org/docs/latest/running-on-yarn.html)) so far.This document will guide you how you can build and configure the environment on 3 types of Spark cluster manager with Apache Zeppelin using [Docker](https://www.docker.com/) scripts.So [install docker](https://docs.docker.com/engine/installation/) on the machine first.## Spark standalone mode[Spark standalone](http://spark.apache.org/docs/latest/spark-standalone.html) is a simple cluster manager included with Spark that makes it easy to set up a cluster.You can simply set up Spark standalone environment with below steps.> **Note :** Since Apache Zeppelin and Spark use same `8080` port for their web UI, you might need to change `zeppelin.server.port` in `conf/zeppelin-site.xml`.### 1. Build Docker fileYou can find docker script files under `scripts/docker/spark-cluster-managers`.```bashcd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_standalonedocker build -t "spark_standalone" .```### 2. Run docker```bashdocker run -it -p 8080:8080 -p 7077:7077 -p 8888:8888 -p 8081:8081 -h sparkmaster --name spark_standalone spark_standalone bash;```Note that `sparkmaster` hostname used here to run docker container should be defined in your `/etc/hosts`.### 3. Configure Spark interpreter in ZeppelinSet Spark master as `spark://:7077` in Zeppelin **Interpreters** setting page.### 4. Run Zeppelin with Spark interpreterAfter running single paragraph with Spark interpreter in Zeppelin, browse `https://:8080` and check whether Spark cluster is running well or not.You can also simply verify that Spark is running well in Docker with below command.```bashps -ef | grep spark```## Spark on YARN modeYou can simply set up [Spark on YARN](http://spark.apache.org/docs/latest/running-on-yarn.html) docker environment with below steps.> **Note :** Since Apache Zeppelin and Spark use same `8080` port for their web UI, you might need to change `zeppelin.server.port` in `conf/zeppelin-site.xml`.### 1. Build Docker fileYou can find docker script files under `scripts/docker/spark-cluster-managers`.```bashcd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_yarn_clusterdocker build -t "spark_yarn" .```### 2. Run docker```bashdocker run -it -p 5000:5000 -p 9000:9000 -p 9001:9001 -p 8088:8088 -p 8042:8042 -p 8030:8030 -p 8031:8031 -p 8032:8032 -p 8033:8033 -p 8080:8080 -p 7077:7077 -p 8888:8888 -p 8081:8081 -p 50010:50010 -p 50075:50075 -p 50020:50020 -p 50070:50070 --name spark_yarn -h sparkmaster spark_yarn bash;```Note that `sparkmaster` hostname used here to run docker container should be defined in your `/etc/hosts`.### 3. Verify running Spark on YARN.You can simply verify the processes of Spark and YARN are running well in Docker with below command.```bashps -ef```You can also check each application web UI for HDFS on `http://:50070/`, YARN on `http://:8088/cluster` and Spark on `http://:8080/`.### 4. Configure Spark interpreter in ZeppelinSet following configurations to `conf/zeppelin-env.sh`.```bashexport MASTER=yarn-clientexport HADOOP_CONF_DIR=[your_hadoop_conf_path]export SPARK_HOME=[your_spark_home_path]````HADOOP_CONF_DIR`(Hadoop configuration path) is defined in `/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf`.Don't forget to set Spark `master` as `yarn-client` in Zeppelin **Interpreters** setting page like below.### 5. Run Zeppelin with Spark interpreterAfter running a single paragraph with Spark interpreter in Zeppelin, browse `http://:8088/cluster/apps` and check Zeppelin application is running well or not.## Spark on Mesos modeYou can simply set up [Spark on Mesos](http://spark.apache.org/docs/latest/running-on-mesos.html) docker environment with below steps.### 1. Build Docker file```bashcd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_mesosdocker build -t "spark_mesos" .```### 2. Run docker```bashdocker run --net=host -it -p 8080:8080 -p 7077:7077 -p 8888:8888 -p 8081:8081 -p 8082:8082 -p 5050:5050 -p 5051:5051 -p 4040:4040 -h sparkmaster --name spark_mesos spark_mesos bash;```Note that `sparkmaster` hostname used here to run docker container should be defined in your `/etc/hosts`.### 3. Verify running Spark on Mesos.You can simply verify the processes of Spark and Mesos are running well in Docker with below command.```bashps -ef```You can also check each application web UI for Mesos on `http://:5050/cluster` and Spark on `http://:8080/`.### 4. Configure Spark interpreter in Zeppelin```bashexport MASTER=mesos://127.0.1.1:5050export MESOS_NATIVE_JAVA_LIBRARY=[PATH OF libmesos.so]export SPARK_HOME=[PATH OF SPARK HOME]```Don't forget to set Spark `master` as `mesos://127.0.1.1:5050` in Zeppelin **Interpreters** setting page like below.### 5. Run Zeppelin with Spark interpreterAfter running a single paragraph with Spark interpreter in Zeppelin, browse `http://:5050/#/frameworks` and check Zeppelin application is running well or not.### Troubleshooting for Spark on Mesos- If you have problem with hostname, use `--add-host` option when executing `dockerrun````## use `--add-host=moby:127.0.0.1` option to resolve## since docker container couldn't resolve `moby`: java.net.UnknownHostException: moby: moby: Name or service not known at java.net.InetAddress.getLocalHost(InetAddress.java:1496) at org.apache.spark.util.Utils$.findLocalInetAddress(Utils.scala:789) at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress$lzycompute(Utils.scala:782) at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress(Utils.scala:782)```- If you have problem with mesos master, try `mesos://127.0.0.1` instead of `mesos://127.0.1.1````I0103 20:17:22.329269 340 sched.cpp:330] New master detected at master@127.0.1.1:5050I0103 20:17:22.330749 340 sched.cpp:341] No credentials provided. Attempting to register without authenticationW0103 20:17:22.333531 340 sched.cpp:736] Ignoring framework registered message because it was sentfrom 'master@127.0.0.1:5050' instead of the leading master 'master@127.0.1.1:5050'W0103 20:17:24.040252 339 sched.cpp:736] Ignoring framework registered message because it was sentfrom 'master@127.0.0.1:5050' instead of the leading master 'master@127.0.1.1:5050'W0103 20:17:26.150250 339 sched.cpp:736] Ignoring framework registered message because it was sentfrom 'master@127.0.0.1:5050' instead of the leading master 'master@127.0.1.1:5050'W0103 20:17:26.737604 339 sched.cpp:736] Ignoring framework registered message because it was sentfrom 'master@127.0.0.1:5050' instead of the leading master 'master@127.0.1.1:5050'W0103 20:17:35.241714 336 sched.cpp:736] Ignoring framework registered message because it was sentfrom 'master@127.0.0.1:5050' instead of the leading master 'master@127.0.1.1:5050'```",
"url": " /setup/deployment/spark_cluster_mode.html",
"group": "setup/deployment",
"excerpt": "This document will guide you how you can build and configure the environment on 3 types of Spark cluster manager(Standalone, Hadoop Yarn, Apache Mesos) with Apache Zeppelin using docker scripts."
}
,
"/setup/deployment/flink_and_spark_cluster.html": {
"title": "Install Zeppelin with Flink and Spark in cluster mode",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Install with Flink and Spark clusterThis tutorial is extremely entry-level. It assumes no prior knowledge of Linux, git, or other tools. If you carefully type what I tell you when I tell you, you should be able to get Zeppelin running.## Installing Zeppelin with Flink and Spark in cluster modeThis tutorial assumes the user has a machine (real or [virtual](https://www.virtualbox.org/wiki/Downloads) with a fresh, minimal installation of [Ubuntu 14.04.3 Server](http://www.ubuntu.com/download/server).**Note:** On the size requirements of the Virtual Machine, some users reported trouble when using the default virtual machine sizes, specifically that the hard drive needed to be at least 16GB- other users did not have this issue.There are many good tutorials on how to install Ubuntu Server on a virtual box, [here is one of them](http://ilearnstack.com/2013/04/13/setting-ubuntu-vm-in-virtualbox/)### Required ProgramsAssuming the minimal install, there are several programs that we will need to install before Zeppelin, Flink, and Spark.- git- openssh-server- OpenJDK 7- Maven 3.1+For git, openssh-server, and OpenJDK 7 we will be using the apt package manager.##### gitFrom the command prompt:```bashsudo apt-get install git```##### openssh-server```bashsudo apt-get install openssh-server```##### OpenJDK 7```bashsudo apt-get install openjdk-7-jdk openjdk-7-jre-lib```*A note for those using Ubuntu 16.04*: To install `openjdk-7` on Ubuntu 16.04, one must add a repository. [Source](http://askubuntu.com/questions/761127/ubuntu-16-04-and-openjdk-7)```bashsudo add-apt-repository ppa:openjdk-r/ppasudo apt-get updatesudo apt-get install openjdk-7-jdk openjdk-7-jre-lib```##### Maven 3.1+Zeppelin requires maven version 3.x. The version available in the repositories at the time of writing is 2.x, so maven must be installed manually.Purge any existing versions of maven.```bashsudo apt-get purge maven maven2```Download the maven 3.3.9 binary.```bashwget "http://www.us.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz"```Unarchive the binary and move to the `/usr/local` directory.```bashtar -zxvf apache-maven-3.3.9-bin.tar.gzsudo mv ./apache-maven-3.3.9 /usr/local```Create symbolic links in `/usr/bin`.```bashsudo ln -s /usr/local/apache-maven-3.3.9/bin/mvn /usr/bin/mvn```### Installing ZeppelinThis provides a quick overview of Zeppelin installation from source, however the reader is encouraged to review the [Zeppelin Installation Guide](../../quickstart/install.html)From the command prompt:Clone Zeppelin.```bashgit clone https://github.com/apache/zeppelin.git```Enter the Zeppelin root directory.```bashcd zeppelin```Package Zeppelin.```bashmvn clean package -DskipTests -Pspark-1.6 -Dflink.version=1.1.3 -Pscala-2.10````-DskipTests` skips build tests- you're not developing (yet), so you don't need to do tests, the clone version *should* build.`-Pspark-1.6` tells maven to build a Zeppelin with Spark 1.6. This is important because Zeppelin has its own Spark interpreter and the versions must be the same.`-Dflink.version=1.1.3` tells maven specifically to build Zeppelin with Flink version 1.1.3.-`-Pscala-2.10` tells maven to build with Scala v2.10.**Note:** You can build against any version of Spark that has a Zeppelin build profile available. The key is to make sure you check out the matching version of Spark to build. At the time of this writing, Spark 1.6 was the most recent Spark version available.**Note:** On build failures. Having installed Zeppelin close to 30 times now, I will tell you that sometimes the build fails for seemingly no reason.As long as you didn't edit any code, it is unlikely the build is failing because of something you did. What does tend to happen, is some dependency that maven is trying to download is unreachable. If your build fails on this step here are some tips:- Don't get discouraged.- Scroll up and read through the logs. There will be clues there.- Retry (that is, run the `mvn clean package -DskipTests -Pspark-1.6` again)- If there were clues that a dependency couldn't be downloaded wait a few hours or even days and retry again. Open source software when compiling is trying to download all of the dependencies it needs, if a server is off-line there is nothing you can do but wait for it to come back.- Make sure you followed all of the steps carefully.- Ask the community to help you. Go [here](http://zeppelin.apache.org/community.html) and join the user mailing list. People are there to help you. Make sure to copy and paste the build output (everything that happened in the console) and include that in your message.Start the Zeppelin daemon.```bashbin/zeppelin-daemon.sh start```Use `ifconfig` to determine the host machine's IP address. If you are not familiar with how to do this, a fairly comprehensive post can be found [here](http://www.cyberciti.biz/faq/how-to-find-out-the-ip-address-assigned-to-eth0-and-display-ip-only/).Open a web-browser on a machine connected to the same network as the host (or in the host operating system if using a virtual machine). Navigate to http://`yourip`:8080, where yourip is the IP address you found in `ifconfig`.See the [Zeppelin tutorial](../../quickstart/tutorial.html) for basic Zeppelin usage. It is also advised that you take a moment to check out the tutorial notebook that is included with each Zeppelin install, and to familiarize yourself with basic notebook functionality.##### Flink TestCreate a new notebook named "Flink Test" and copy and paste the following code.```scala%flink // let Zeppelin know what interpreter to use.val text = benv.fromElements("In the time of chimpanzees, I was a monkey", // some lines of text to analyze"Butane in my veins and I'm out to cut the junkie","With the plastic eyeballs, spray paint the vegetables","Dog food stalls with the beefcake pantyhose","Kill the headlights and put it in neutral","Stock car flamin' with a loser in the cruise control","Baby's in Reno with the Vitamin D","Got a couple of couches, sleep on the love seat","Someone came in sayin' I'm insane to complain","About a shotgun wedding and a stain on my shirt","Don't believe everything that you breathe","You get a parking violation and a maggot on your sleeve","So shave your face with some mace in the dark","Savin' all your food stamps and burnin' down the trailer park","Yo, cut it")/* The meat and potatoes: this tells Flink to iterate through the elements, in this case strings, transform the string to lower case and split the string at white space into individual words then finally aggregate the occurrence of each word. This creates the count variable which is a list of tuples of the form (word, occurances)counts.collect().foreach(println(_)) // execute the script and print each element in the counts list*/val counts = text.flatMap{ _.toLowerCase.split("W+") }.map { (_,1) }.groupBy(0).sum(1)counts.collect().foreach(println(_)) // execute the script and print each element in the counts list```Run the code to make sure the built-in Zeppelin Flink interpreter is working properly.##### Spark TestCreate a new notebook named "Spark Test" and copy and paste the following code.```scala%spark // let Zeppelin know what interpreter to use.val text = sc.parallelize(List("In the time of chimpanzees, I was a monkey", // some lines of text to analyze"Butane in my veins and I'm out to cut the junkie","With the plastic eyeballs, spray paint the vegetables","Dog food stalls with the beefcake pantyhose","Kill the headlights and put it in neutral","Stock car flamin' with a loser in the cruise control","Baby's in Reno with the Vitamin D","Got a couple of couches, sleep on the love seat","Someone came in sayin' I'm insane to complain","About a shotgun wedding and a stain on my shirt","Don't believe everything that you breathe","You get a parking violation and a maggot on your sleeve","So shave your face with some mace in the dark","Savin' all your food stamps and burnin' down the trailer park","Yo, cut it"))/* The meat and potatoes: this tells spark to iterate through the elements, in this case strings, transform the string to lower case and split the string at white space into individual words then finally aggregate the occurrence of each word. This creates the count variable which is a list of tuples of the form (word, occurances)*/val counts = text.flatMap { _.toLowerCase.split("W+") } .map { (_,1) } .reduceByKey(_ + _)counts.collect().foreach(println(_)) // execute the script and print each element in the counts list```Run the code to make sure the built-in Zeppelin Flink interpreter is working properly.Finally, stop the Zeppelin daemon. From the command prompt run:```bashbin/zeppelin-daemon.sh stop```### Installing Clusters##### Flink Cluster###### Download BinariesBuilding from source is recommended where possible, for simplicity in this tutorial we will download Flink and Spark Binaries.To download the Flink Binary use `wget````bashwget "http://mirror.cogentco.com/pub/apache/flink/flink-1.1.3/flink-1.1.3-bin-hadoop24-scala_2.10.tgz"tar -xzvf flink-1.1.3-bin-hadoop24-scala_2.10.tgz```This will download Flink 1.1.3, compatible with Hadoop 2.4. You do not have to install Hadoop for this binary to work, but if you are using Hadoop, please change `24` to your appropriate version.Start the Flink Cluster.```bashflink-1.1.3/bin/start-cluster.sh```###### Building From sourceIf you wish to build Flink from source, the following will be instructive. Note that if you have downloaded and used the binary version this should be skipped. The changing nature of build tools and versions across platforms makes this section somewhat precarious. For example, Java8 and Maven 3.0.3 are recommended for building Flink, which are not recommended for Zeppelin at the time of writing. If the user wishes to attempt to build from source, this section will provide some reference. If errors are encountered, please contact the Apache Flink community.See the [Flink Installation guide](https://github.com/apache/flink/blob/master/README.md) for more detailed instructions.Return to the directory where you have been downloading, this tutorial assumes that is `$HOME`. Clone Flink, check out release-1.1.3-rc2, and build.```bashcd $HOMEgit clone https://github.com/apache/flink.gitcd flinkgit checkout release-1.1.3-rc2mvn clean install -DskipTests```Start the Flink Cluster in stand-alone mode```bashbuild-target/bin/start-cluster.sh```###### Ensure the cluster is upIn a browser, navigate to http://`yourip`:8082 to see the Flink Web-UI. Click on 'Task Managers' in the left navigation bar. Ensure there is at least one Task Manager present.If no task managers are present, restart the Flink cluster with the following commands:(if binaries)```bashflink-1.1.3/bin/stop-cluster.shflink-1.1.3/bin/start-cluster.sh```(if built from source)```bashbuild-target/bin/stop-cluster.shbuild-target/bin/start-cluster.sh```##### Spark 1.6 Cluster###### Download BinariesBuilding from source is recommended where possible, for simplicity in this tutorial we will download Flink and Spark Binaries.Using binaries is alsoTo download the Spark Binary use `wget````bashwget "http://d3kbcqa49mib13.cloudfront.net/spark-1.6.3-bin-hadoop2.6.tgz"tar -xzvf spark-1.6.3-bin-hadoop2.6.tgzmv spark-1.6.3-bin-hadoop2.6 spark```This will download Spark 1.6.3, compatible with Hadoop 2.6. You do not have to install Hadoop for this binary to work, but if you are using Hadoop, please change `2.6` to your appropriate version.###### Building From sourceSpark is an extraordinarily large project, which takes considerable time to download and build. It is also prone to build failures for similar reasons listed in the Flink section. If the user wishes to attempt to build from source, this section will provide some reference. If errors are encountered, please contact the Apache Spark community.See the [Spark Installation](https://github.com/apache/spark/blob/master/README.md) guide for more detailed instructions.Return to the directory where you have been downloading, this tutorial assumes that is $HOME. Clone Spark, check out branch-1.6, and build.**Note:** Recall, we're only checking out 1.6 because it is the most recent Spark for which a Zeppelin profile exists at the time of writing. You are free to check out other version, just make sure you build Zeppelin against the correct version of Spark. However if you use Spark 2.0, the word count example will need to be changed as Spark 2.0 is not compatible with the following examples.```bashcd $HOME```Clone, check out, and build Spark version 1.6.x.```bashgit clone https://github.com/apache/spark.gitcd sparkgit checkout branch-1.6mvn clean package -DskipTests```###### Start the Spark clusterReturn to the `$HOME` directory.```bashcd $HOME```Start the Spark cluster in stand alone mode, specifying the webui-port as some port other than 8080 (the webui-port of Zeppelin).```bashspark/sbin/start-master.sh --webui-port 8082```**Note:** Why `--webui-port 8082`? There is a digression toward the end of this document that explains this.Open a browser and navigate to http://`yourip`:8082 to ensure the Spark master is running.Toward the top of the page there will be a *URL*: spark://`yourhost`:7077. Note this URL, the Spark Master URI, it will be needed in subsequent steps.Start the slave using the URI from the Spark master WebUI:```bashspark/sbin/start-slave.sh spark://yourhostname:7077```Return to the root directory and start the Zeppelin daemon.```bashcd $HOMEzeppelin/bin/zeppelin-daemon.sh start```##### Configure InterpretersOpen a web browser and go to the Zeppelin web-ui at http://yourip:8080.Now go back to the Zeppelin web-ui at http://`yourip`:8080 and this time click on *anonymous* at the top right, which will open a drop-down menu, select *Interpreters* to enter interpreter configuration.In the Spark section, click the edit button in the top right corner to make the property values editable (looks like a pencil).The only field that needs to be edited in the Spark interpreter is the master field. Change this value from `local[*]` to the URL you used to start the slave, mine was `spark://ubuntu:7077`.Click *Save* to update the parameters, and click *OK* when it asks you about restarting the interpreter.Now scroll down to the Flink section. Click the edit button and change the value of *host* from `local` to `localhost`. Click *Save* again.Reopen the examples and execute them again (I.e. you need to click the play button at the top of the screen, or the button on the paragraph .You should be able check the Flink and Spark webuis (at something like http://`yourip`:8081, http://`yourip`:8082, http://`yourip`:8083) and see that jobs have been run against the clusters.**Digression** Sorry to be vague and use terms such as 'something like', but exactly what web-ui is at what port is going to depend on what order you started things. What is really going on here is you are pointing your browser at specific ports, namely 8081, 8082, and 8083. Flink and Spark all want to put their web-ui on port 8080, but are well behaved and will take the next port available. Since Zeppelin started first, it will get port 8080. When Flink starts (assuming you started Flink first), it will try to bind to port 8080, see that it is already taken, and go to the next one available, hopefully 8081. Spark has a webui for the master and the slave, so when they start they will try to bind to 8080 already taken by Zeppelin), then 8081 (already taken by Flink's webui), then 8082. If everything goes smoothy and you followed the directions precisely, the webuis should be 8081 and 8082. It *is* possible to specify the port you want the webui to bind to (at the command line by passing the `--webui-port ` flag when you start the Flink and Spark, where `` is the port you want to see that webui on. You can also set the default webui port of Spark and Flink (and Zeppelin) in the configuration files, but this is a tutorial for novices and slightly out of scope.### Next StepsCheck out the [tutorial](../../quickstart/tutorial.html) for more cool things you can do with your new toy, ask questions and contribute! Every little bit helps.",
"url": " /setup/deployment/flink_and_spark_cluster.html",
"group": "setup/deployment",
"excerpt": ""
}
,
"/setup/deployment/cdh.html": {
"title": "Apache Zeppelin on CDH",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Apache Zeppelin on CDH### 1. Import Cloudera QuickStart Docker image>[Cloudera](http://www.cloudera.com/) has officially provided CDH Docker Hub in their own container. Please check [this guide page](http://www.cloudera.com/documentation/enterprise/latest/topics/quickstart_docker_container.html#cloudera_docker_container) for more information.You can import the Docker image by pulling it from Cloudera Docker Hub.```bashdocker pull cloudera/quickstart:latest```### 2. Run docker```bashdocker run -it -p 80:80 -p 4040:4040 -p 8020:8020 -p 8022:8022 -p 8030:8030 -p 8032:8032 -p 8033:8033 -p 8040:8040 -p 8042:8042 -p 8088:8088 -p 8480:8480 -p 8485:8485 -p 8888:8888 -p 9083:9083 -p 10020:10020 -p 10033:10033 -p 18088:18088 -p 19888:19888 -p 25000:25000 -p 25010:25010 -p 25020:25020 -p 50010:50010 -p 50020:50020 -p 50070:50070 -p 50075:50075 -h quickstart.cloudera --privileged=true agitated_payne_backup /usr/bin/docker-quickstart;```### 3. Verify running CDHTo verify the application is running well, check the web UI for HDFS on `http://:50070/` and YARN on `http://:8088/cluster`.### 4. Configure Spark interpreter in ZeppelinSet following configurations to `conf/zeppelin-env.sh`.```bashexport MASTER=yarn-clientexport HADOOP_CONF_DIR=[your_hadoop_conf_path]export SPARK_HOME=[your_spark_home_path]````HADOOP_CONF_DIR`(Hadoop configuration path) is defined in `/scripts/docker/spark-cluster-managers/cdh/hdfs_conf`.Don't forget to set Spark `master` as `yarn-client` in Zeppelin **Interpreters** setting page like below.### 5. Run Zeppelin with Spark interpreterAfter running a single paragraph with Spark interpreter in Zeppelin,browse `http://:8088/cluster/apps` to check Zeppelin application is running well or not.",
"url": " /setup/deployment/cdh.html",
"group": "setup/deployment",
"excerpt": "This document will guide you how you can build and configure the environment on CDH with Apache Zeppelin using docker scripts."
}
,
"/setup/deployment/docker.html": {
"title": "Apache Zeppelin Releases Docker Images",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Docker Image for Apache Zeppelin Releases ## Overview This document contains instructions about making docker containers for Zeppelin. It mainly provides guidance into how to create, publish and run docker images for zeppelin releases.## Quick Start### Installing DockerYou need to [install docker](https://docs.docker.com/engine/installation/) on your machine.### Running docker image```bashdocker run -p 8080:8080 -e ZEPPELIN_ADDR='0.0.0.0' --rm --name zeppelin apache/zeppelin: ```* Zeppelin will run at `http://localhost:8080`.If you want to specify `logs` and `notebook` dir, ```bashdocker run -p 8080:8080 --rm -v $PWD/logs:/logs -v $PWD/notebook:/notebook -e ZEPPELIN_LOG_DIR='/logs' -e ZEPPELIN_NOTEBOOK_DIR='/notebook' --name zeppelin apache/zeppelin: # e.g '0.7.1'```### Building dockerfile locally```bashcd $ZEPPELIN_HOMEcd scripts/docker/zeppelin/bindocker build -t my-zeppelin:my-tag ./```",
"url": " /setup/deployment/docker.html",
"group": "setup/deployment",
"excerpt": "This document contains instructions about making docker containers for Zeppelin. It mainly provides guidance into how to create, publish and run docker images for zeppelin releases."
}
,
"/setup/deployment/yarn_install.html": {
"title": "Install Zeppelin to connect with existing YARN cluster",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}## IntroductionThis page describes how to pre-configure a bare metal node, configure Zeppelin and connect it to existing YARN cluster running Hortonworks flavour of Hadoop. It also describes steps to configure Spark interpreter of Zeppelin.## Prepare Node### Zeppelin user (Optional)This step is optional, however its nice to run Zeppelin under its own user. In case you do not like to use Zeppelin (hope not) the user could be deleted along with all the packages that were installed for Zeppelin, Zeppelin binary itself and associated directories.Create a zeppelin user and switch to zeppelin user or if zeppelin user is already created then login as zeppelin.```bashuseradd zeppelinsu - zeppelinwhoami```Assuming a zeppelin user is created then running whoami command must return```bashzeppelin```Its assumed in the rest of the document that zeppelin user is indeed created and below installation instructions are performed as zeppelin user.### List of Prerequisites * CentOS 6.x, Mac OSX, Ubuntu 14.X * Java 1.7 * Hadoop client * Spark * Internet connection is required.It's assumed that the node has CentOS 6.x installed on it. Although any version of Linux distribution should work fine.#### Hadoop clientZeppelin can work with multiple versions & distributions of Hadoop. A complete list is available [here](https://github.com/apache/zeppelin#build). This document assumes Hadoop 2.7.x client libraries including configuration files are installed on Zeppelin node. It also assumes /etc/hadoop/conf contains various Hadoop configuration files. The location of Hadoop configuration files may vary, hence use appropriate location.```bashhadoop versionHadoop 2.7.1.2.3.1.0-2574Subversion git@github.com:hortonworks/hadoop.git -r f66cf95e2e9367a74b0ec88b2df33458b6cff2d0Compiled by jenkins on 2015-07-25T22:36ZCompiled with protoc 2.5.0From source with checksum 54f9bbb4492f92975e84e390599b881dThis command was run using /usr/hdp/2.3.1.0-2574/hadoop/lib/hadoop-common-2.7.1.2.3.1.0-2574.jar```#### SparkSpark is supported out of the box and to take advantage of this, you need to Download appropriate version of Spark binary packages from [Spark Download page](http://spark.apache.org/downloads.html) and unzip it.Zeppelin can work with multiple versions of Spark. A complete list is available [here](https://github.com/apache/zeppelin#build).This document assumes Spark 1.6.0 is installed at /usr/lib/spark.> Note: Spark should be installed on the same node as Zeppelin.> Note: Spark's pre-built package for CDH 4 doesn't support yarn.#### ZeppelinCheckout source code from [git://git.apache.org/zeppelin.git](https://github.com/apache/zeppelin.git) or download binary package from [Download page](https://zeppelin.apache.org/download.html).You can refer [Install](install.html) page for the details.This document assumes that Zeppelin is located under `/home/zeppelin/zeppelin`.## Zeppelin ConfigurationZeppelin configuration needs to be modified to connect to YARN cluster. Create a copy of zeppelin environment shell script.```bashcp /home/zeppelin/zeppelin/conf/zeppelin-env.sh.template /home/zeppelin/zeppelin/conf/zeppelin-env.sh```Set the following properties```bashexport JAVA_HOME="/usr/java/jdk1.7.0_79"export HADOOP_CONF_DIR="/etc/hadoop/conf"export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"export SPARK_HOME="/usr/lib/spark"```As /etc/hadoop/conf contains various configurations of YARN cluster, Zeppelin can now submit Spark/Hive jobs on YARN cluster form its web interface. The value of hdp.version is set to 2.3.1.0-2574. This can be obtained by running the following command```bashhdp-select status hadoop-client | sed 's/hadoop-client - (.*)/1/'# It returned 2.3.1.0-2574```## Start/Stop### Start Zeppelin```bashcd /home/zeppelin/zeppelinbin/zeppelin-daemon.sh start```After successful start, visit http://[zeppelin-server-host-name]:8080 with your web browser.### Stop Zeppelin```bashbin/zeppelin-daemon.sh stop```## InterpreterZeppelin provides various distributed processing frameworks to process data that ranges from Spark, JDBC, Ignite and Lens to name a few. This document describes to configure JDBC & Spark interpreters.### HiveZeppelin supports Hive through JDBC interpreter. You might need the information to use Hive and can find in your hive-site.xmlOnce Zeppelin server has started successfully, visit http://[zeppelin-server-host-name]:8080 with your web browser. Click on Interpreter tab next to Notebook dropdown. Look for Hive configurations and set them appropriately. Set them as per Hive installation on YARN cluster.Click on Save button. Once these configurations are updated, Zeppelin will prompt you to restart the interpreter. Accept the prompt and the interpreter will reload the configurations.### SparkIt was assumed that 1.6.0 version of Spark is installed at /usr/lib/spark. Look for Spark configurations and click edit button to add the following properties Property Name Property Value Remarks master yarn-client In yarn-client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN. spark.driver.extraJavaOptions -Dhdp.version=2.3.1.0-2574 spark.yarn.am.extraJavaOptions -Dhdp.version=2.3.1.0-2574 Click on Save button. Once these configurations are updated, Zeppelin will prompt you to restart the interpreter. Accept the prompt and the interpreter will reload the configurations.Spark & Hive notebooks can be written with Zeppelin now. The resulting Spark & Hive jobs will run on configured YARN cluster.## DebugZeppelin does not emit any kind of error messages on web interface when notebook/paragraph is run. If a paragraph fails it only displays ERROR. The reason for failure needs to be looked into log files which is present in logs directory under zeppelin installation base directory. Zeppelin creates a log file for each kind of interpreter.```bash[zeppelin@zeppelin-3529 logs]$ pwd/home/zeppelin/zeppelin/logs[zeppelin@zeppelin-3529 logs]$ ls -ltotal 844-rw-rw-r-- 1 zeppelin zeppelin 14648 Aug 3 14:45 zeppelin-interpreter-hive-zeppelin-zeppelin-3529.log-rw-rw-r-- 1 zeppelin zeppelin 625050 Aug 3 16:05 zeppelin-interpreter-spark-zeppelin-zeppelin-3529.log-rw-rw-r-- 1 zeppelin zeppelin 200394 Aug 3 21:15 zeppelin-zeppelin-zeppelin-3529.log-rw-rw-r-- 1 zeppelin zeppelin 16162 Aug 3 14:03 zeppelin-zeppelin-zeppelin-3529.out[zeppelin@zeppelin-3529 logs]$```",
"url": " /setup/deployment/yarn_install.html",
"group": "setup/deployment",
"excerpt": "This page describes how to pre-configure a bare metal node, configure Apache Zeppelin and connect it to existing YARN cluster running Hortonworks flavour of Hadoop."
}
,
"/usage/interpreter/overview.html": {
"title": "Interpreter in Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Interpreter in Apache Zeppelin## OverviewIn this section, we will explain about the role of interpreters, interpreters group and interpreter settings in Zeppelin.The concept of Zeppelin interpreter allows any language/data-processing-backend to be plugged into Zeppelin.Currently, Zeppelin supports many interpreters such as Scala ( with Apache Spark ), Python ( with Apache Spark ), Spark SQL, JDBC, Markdown, Shell and so on.## What is Zeppelin interpreter?Zeppelin Interpreter is a plug-in which enables Zeppelin users to use a specific language/data-processing-backend. For example, to use Scala code in Zeppelin, you need `%spark` interpreter.When you click the ```+Create``` button in the interpreter page, the interpreter drop-down list box will show all the available interpreters on your server.## What is interpreter setting?Zeppelin interpreter setting is the configuration of a given interpreter on Zeppelin server. For example, the properties are required for hive JDBC interpreter to connect to the Hive server.Properties are exported as environment variables when property name is consisted of upper characters, numbers and underscore ([A-Z_0-9]). Otherwise set properties as JVM property. You may use parameters from the context of interpreter by add #{contextParameterName} in value, parameter can be of the following types: string, number, boolean.###### Context parameters Name Type user string noteId string replName string className string If context parameter is null then replaced by empty string.Each notebook can be bound to multiple Interpreter Settings using setting icon on upper right corner of the notebook.## What is interpreter group?Every Interpreter is belonged to an **Interpreter Group**. Interpreter Group is a unit of start/stop interpreter.By default, every interpreter is belonged to a single group, but the group might contain more interpreters. For example, Spark interpreter group is including Spark support, pySpark, Spark SQL and the dependency loader.Technically, Zeppelin interpreters from the same group are running in the same JVM. For more information about this, please checkout [here](../development/writing_zeppelin_interpreter.html).Each interpreters is belonged to a single group and registered together. All of their properties are listed in the interpreter setting like below image.## Interpreter binding modeEach Interpreter Setting can choose one of 'shared', 'scoped', 'isolated' interpreter binding mode.In 'shared' mode, every notebook bound to the Interpreter Setting will share the single Interpreter instance. In 'scoped' mode, each notebook will create new Interpreter instance in the same interpreter process. In 'isolated' mode, each notebook will create new Interpreter process.For more information, check [Interpreter Binding Mode](./interpreter_binding_mode.html).## Connecting to the existing remote interpreterZeppelin users can start interpreter thread embedded in their service. This will provide flexibility to user to start interpreter on remote host. To start interpreter along with your service you have to create an instance of ``RemoteInterpreterServer`` and start it as follows:```javaRemoteInterpreterServer interpreter=new RemoteInterpreterServer(3678); // Here, 3678 is the port on which interpreter will listen. interpreter.start();```The above code will start interpreter thread inside your process. Once the interpreter is started you can configure zeppelin to connect to RemoteInterpreter by checking **Connect to existing process** checkbox and then provide **Host** and **Port** on which interpreter process is listening as shown in the image below:## PrecodeSnippet of code (language of interpreter) that executes after initialization of the interpreter depends on [Binding mode](#interpreter-binding-mode). To configure add parameter with class of interpreter (`zeppelin..precode`) except JDBCInterpreter ([JDBC precode](../../interpreter/jdbc.html#usage-precode)). ## Interpreter Lifecycle ManagementBefore 0.8.0, Zeppelin don't have lifecycle management on interpreter. User have to shutdown interpreters explicitly via UI. Starting from 0.8.0, Zeppelin provides a new interface`LifecycleManager` to control the lifecycle of interpreters. For now, there're 2 implementations: `NullLifecycleManager` and `TimeoutLifecycleManager` which is default. `NullLifecycleManager` will do nothing,user need to control the lifecycle of interpreter by themselves as before. `TimeoutLifecycleManager` will shutdown interpreters after interpreter idle for a while. By default, the idle threshold is 1 hour.User can change it via `zeppelin.interpreter.lifecyclemanager.timeout.threshold`. `TimeoutLifecycleManager` is the default lifecycle manager, user can change it via `zeppelin.interpreter.lifecyclemanager.class`.## Generic ConfInterpreterZeppelin's interpreter setting is shared by all users and notes, if you want to have different setting you have to create new interpreter, e.g. you can create `spark_jar1` for running spark with dependency jar1 and `spark_jar2` for running spark with dependency jar2.This approach works, but not so convenient. `ConfInterpreter` can provide more fine-grained control on interpreter setting and more flexibility. `ConfInterpreter` is a generic interpreter that could be used by any interpreters. The input format should be property file format.It can be used to make custom setting for any interpreter. But it requires to run before interpreter process launched. And when interpreter process is launched is determined by interpreter mode setting.So users needs to understand the ([interpreter mode setting ](../usage/interpreter/interpreter_bindings_mode.html) of Zeppelin and be aware when interpreter process is launched. E.g. If we set spark interpreter setting as isolated per note. Under this setting, each note will launch one interpreter process. In this scenario, user need to put `ConfInterpreter` as the first paragraph as the below example. Otherwise the customized setting can not be applied (Actually it would report ERROR)## Interpreter Process RecoveryBefore 0.8.0, shutting down Zeppelin also mean to shutdown all the running interpreter processes. Usually admin will shutdown Zeppelin server for maintenance or upgrade, but don't want to shut down the running interpreter processes.In such cases, interpreter process recovery is necessary. Starting from 0.8.0, user can enable interpreter process recovering via setting `zeppelin.recovery.storage.class` as `org.apache.zeppelin.interpreter.recovery.FileSystemRecoveryStorage` or other implementations if available in future, by default it is `org.apache.zeppelin.interpreter.recovery.NullRecoveryStorage` which means recovery is not enabled. Enable recover means shutting down Zeppelin would not terminating interpreter process,and when Zeppelin is restarted, it would try to reconnect to the existing running interpreter processes. If you want to kill all the interpreter processes after terminating Zeppelin even when recovery is enabled, you can run `bin/stop-interpreter.sh` ## Credential InjectionCredentials from the credential manager can be injected into Notebooks. Credential injection works by replacing the following patterns in Notebooks with matching credentials for the Credential Manager: `{user.CREDENTIAL_ENTITY}` and `{password.CREDENTIAL_ENTITY}`. However, credential injection must be enabled per Interpreter, by adding a boolean `injectCredentials` setting in the Interpreters configuration. Injected passwords are removed from Notebook output to prevent accidentally leaking passwords.**Credential Injection Setting****Credential Entry Example****Credential Injection Example**```val password = "{password.SOME_CREDENTIAL_ENTITY}"val username = "{user.SOME_CREDENTIAL_ENTITY}"```",
"url": " /usage/interpreter/overview.html",
"group": "usage/interpreter",
"excerpt": "This document explains about the role of interpreters, interpreters group and interpreter settings in Apache Zeppelin. The concept of Zeppelin interpreter allows any language/data-processing-backend to be plugged into Zeppelin."
}
,
"/usage/interpreter/user_impersonation.html": {
"title": "Impersonation",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# ImpersonationUser impersonation enables to run zeppelin interpreter process as a web frontend user## Setup#### 1. Enable Shiro auth in `conf/shiro.ini````[users]user1 = password1, role1user2 = password2, role2```#### 2. Enable password-less ssh for the user you want to impersonate (say user1).```bashadduser user1#ssh-keygen (optional if you don't already have generated ssh-key.ssh user1@localhost mkdir -p .sshcat ~/.ssh/id_rsa.pub | ssh user1@localhost 'cat >> .ssh/authorized_keys'```Alternatively instead of password-less, user can override ZEPPELIN_IMPERSONATE_CMD in zeppelin-env.sh```bashexport ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c '```#### 4. Restart zeppelin server.```bash# for OSX, linuxbin/zeppelin-daemon restart# for windowsbinzeppelin.cmd```#### 5. Configure impersonation for interpreter Go to interpreter setting page, and enable "User Impersonate" in any of the interpreter (in my example its shell interpreter)#### 6. Test with a simple paragraph```bash%shwhoami```Note that usage of "User Impersonate" option will enable Spark interpreter to use `--proxy-user` option with current user by default. If you want to disable `--proxy-user` option, then refer to `ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER` variable in `conf/zeppelin-env.sh`",
"url": " /usage/interpreter/user_impersonation.html",
"group": "usage/interpreter",
"excerpt": "Set up zeppelin interpreter process as web front end user."
}
,
"/usage/interpreter/dependency_management.html": {
"title": "Dependency Management for Interpreter",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}## Dependency Management for InterpreterYou can include external libraries to interpreter by setting dependencies in interpreter menu.When your code requires external library, instead of doing download/copy/restart Zeppelin, you can easily do following jobs in this menu. * Load libraries recursively from Maven repository * Load libraries from local filesystem * Add additional maven repository * Automatically add libraries to SparkCluster Load Dependencies to Interpreter Click 'Interpreter' menu in navigation bar. Click 'edit' button of the interpreter which you want to load dependencies to. Fill artifact and exclude field to your needs. You can enter not only groupId:artifactId:version but also local file in artifact field. Press 'Save' to restart the interpreter with loaded libraries. Add repository for dependency resolving Press icon in 'Interpreter' menu on the top right side. It will show you available repository lists. If you need to resolve dependencies from other than central maven repository or local ~/.m2 repository, hit icon next to repository lists. Fill out the form and click 'Add' button, then you will be able to see that new repository is added. Optionally, if you are behind a corporate firewall, you can specify also all proxy settings so that Zeppelin can download the dependencies using the given credentials ",
"url": " /usage/interpreter/dependency_management.html",
"group": "usage/interpreter",
"excerpt": "Include external libraries to Apache Spark Interpreter by setting dependencies in interpreter menu."
}
,
"/usage/interpreter/execution_hooks.html": {
"title": "Interpreter Execution Hooks (Experimental)",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Interpreter Execution Hooks (Experimental)## OverviewApache Zeppelin allows for users to specify additional code to be executed by an interpreter at pre and post-paragraph code execution.This is primarily useful if you need to run the same set of code for all of the paragraphs within your notebook at specific times.Currently, this feature is only available for the spark and pyspark interpreters.To specify your hook code, you may use `z.registerHook()`. For example, enter the following into one paragraph:```python%pysparkz.registerHook("post_exec", "print 'This code should be executed before the parapgraph code!'")z.registerHook("pre_exec", "print 'This code should be executed after the paragraph code!'")```These calls will not take into effect until the next time you run a paragraph. In another paragraph, enter```python%pysparkprint "This code should be entered into the paragraph by the user!"```The output should be:```This code should be executed before the paragraph code!This code should be entered into the paragraph by the user!This code should be executed after the paragraph code!```If you ever need to know the hook code, use `z.getHook()`:```python%pysparkprint z.getHook("post_exec")print 'This code should be executed after the paragraph code!'```Any call to `z.registerHook()` will automatically overwrite what was previously registered.To completely unregister a hook event, use `z.unregisterHook(eventCode)`.Currently only `"post_exec"` and `"pre_exec"` are valid event codes for the Zeppelin Hook Registry system.Finally, the hook registry is internally shared by other interpreters in the same group.This would allow for hook code for one interpreter REPL to be set by another as follows:```scala%sparkz.unregisterHook("post_exec", "pyspark")```The API is identical for both the spark (scala) and pyspark (python) implementations.### CaveatsCalls to `z.registerHook("pre_exec", ...)` should be made with care. If there are errors in your specified hook code, this will cause the interpreter REPL to become unable to execute any code pass the pre-execute stage making it impossible for direct calls to `z.unregisterHook()` to take into effect. Current workarounds include calling `z.unregisterHook()` from a different interpreter REPL in the same interpreter group (see above) or manually restarting the interpreter group in the UI. ",
"url": " /usage/interpreter/execution_hooks.html",
"group": "manual",
"excerpt": "Apache Zeppelin allows for users to specify additional code to be executed by an interpreter at pre and post-paragraph code execution."
}
,
"/usage/interpreter/interpreter_binding_mode.html": {
"title": "Interpreter Binding Mode",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Interpreter Binding Mode ## OverviewInterpreter Process is a JVM process that communicates with Zeppelin daemon using thrift. Each interpreter process has a single interpreter group, and this interpreter group can have one or more instances of an interpreter.(See [here](../../development/writing_zeppelin_interpreter.html) to understand more about its internal structure.) Zeppelin provides 3 different modes to run interpreter process: **shared**, **scoped** and **isolated**. Also, the user can specify the scope of these modes: **per** user or **per note**.These 3 modes give flexibility to fit Zeppelin into any type of use cases.In this documentation, we mainly discuss the **per note** scope in combination with the **shared**, **scoped** and **isolated** modes.## Shared Mode In **Shared** mode, single JVM process and a single session serves all notes. As a result, `note A` can access variables (e.g python, scala, ..) directly created from other notes.. ## Scoped Mode In **Scoped** mode, Zeppelin still runs a single interpreter JVM process but, in the case of per note scope, each note runs in its own dedicated session. (Note it is still possible to share objects between these notes via [ResourcePool](../../interpreter/spark.html#object-exchange)) ## Isolated Mode **Isolated** mode runs a separate interpreter process for each note in the case of **per note** scope. So, each note has an absolutely isolated session. (But it is still possible to share objects via [ResourcePool](../../interpreter/spark.html#object-exchange)) ## Which mode should I use?Mode | Each notebook... | Benefits | Disadvantages | Sharing objects--- | --- | --- | --- | ---**shared** | Shares a single session in a single interpreter process (JVM) | Low resource utilization and it's easy to share data between notebooks | All notebooks are affected if the interpreter process dies | Can share directly**scoped** | Has its own session in the same interpreter process (JVM) | Less resource utilization than isolated mode | All notebooks are affected if the interpreter process dies | Can't share directly, but it's possible to share objects via [ResourcePool](../../interpreter/spark.html#object-exchange)) **isolated** | Has its own Interpreter Process | One notebook is not affected directly by other notebooks (**per note**) | Can't share data between notebooks easily (**per note**) | Can't share directly, but it's possible to share objects via [ResourcePool](../../interpreter/spark.html#object-exchange)) In the case of the **per user** scope (available in a multi-user environment), Zeppelin manages interpreter sessions on a per user basis rather than a per note basis. For example: - In **scoped + per user** mode, `User A`'s notes **might** be affected by `User B`'s notes. (e.g JVM dies, ...) Because all notes are running on the same JVM- On the other hand, **isolated + per user** mode, `User A`'s notes will not be affected by others' notes which running on separated JVMsEach Interpreter implementation may have different characteristics depending on the back end system that they integrate. And 3 interpreter modes can be used differently.Let’s take a look how Spark Interpreter implementation uses these 3 interpreter modes with **per note** scope, as an example.Spark Interpreter implementation includes 4 different interpreters in the group: Spark, SparkSQL, Pyspark and SparkR. SparkInterpreter instance embeds Scala REPL for interactive Spark API execution. In Shared mode, a SparkContext and a Scala REPL is being shared among all interpreters in the group. So every note will be sharing single SparkContext and single Scala REPL. In this mode, if `Note A` defines variable ‘a’ then `Note B` not only able to read variable ‘a’ but also able to override the variable. In Scoped mode, each note has its own Scala REPL. So variable defined in a note can not be read or overridden in another note. However, a single SparkContext still serves all the sessions.And all the jobs are submitted to this SparkContext and the fair scheduler schedules the jobs.This could be useful when user does not want to share Scala session, but want to keep single Spark application and leverage its fair scheduler.In Isolated mode, each note has its own SparkContext and Scala REPL. ",
"url": " /usage/interpreter/interpreter_binding_mode.html",
"group": "usage/interpreter",
"excerpt": ""
}
,
"/usage/interpreter/dynamic_loading.html": {
"title": "Dynamic Interpreter Loading using REST API",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Dynamic Interpreter Loading using REST APIApache Zeppelin provides pluggable interpreter architecture which results in a wide and variety of the supported backend system. In this section, we will introduce **Dynamic interpreter loading** using **REST API**. This concept actually comes from [Zeppelin Helium Proposal](https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal).Before we start, if you are not familiar with the concept of **Zeppelin interpreter**, you can check out [Overview: Zeppelin Interpreter](./overview.html) first.## Overview In the past, Zeppelin was loading interpreter binaries from `/interpreter/[interpreter_name]` directory. They were configured by `zeppelin.interpreters` property in `conf/zeppelin-site.xml` or `ZEPPELIN_INTERPRETERS` env variables in `conf/zeppelin-env.sh`. They were loaded on Zeppelin server startup and stayed alive until the server was stopped.In order to simplify using 3rd party interpreters, we changed this way to **dynamically** load interpreters from **Maven Repository** using **REST API**. Hopefully, the picture below will help you to understand the process. ## Load & Unload Interpreters Using REST API### Load You can **load** interpreters located in Maven repository using REST API, like this:(Maybe, you are unfamiliar with `[interpreter_group_name]` or `[interpreter_name]`. If so, please checkout [Overview: Zeppelin Interpreter](./overview.html) again.)```http://[zeppelin-server]:[zeppelin-port]/api/interpreter/load/[interpreter_group_name]/[interpreter_name]```The Restful method will be **POST**. And the parameters you need are: 1. **Artifact:** Maven artifact ( groupId:artifactId:version ) 2. **Class Name:** Package name + Interpreter class name 3. **Repository ( optional ):** Additional maven repository addressFor example, if you want to load `markdown` interpreter to your Zeppelin, the parameters and URL you need may look like:```http://127.0.0.1:8080/api/interpreter/load/md/markdown``````json{ "artifact": "org.apache.zeppelin:zeppelin-markdown:0.6.0-SNAPSHOT", "className": "org.apache.zeppelin.markdown.Markdown", "repository": { "url": "http://dl.bintray.com/spark-packages/maven", "snapshot": false }}```The meaning of each parameters is: 1. **Artifact** - groupId: org.apache.zeppelin - artifactId: zeppelin-markdown - version: 0.6.0-SNAPSHOT 2. **Class Name** - Package Name: org.apache.zeppelin - Interpreter Class Name: markdown.Markdown 3. **Repository ( optional )** - Url: http://dl.bintray.com/spark-packages/maven - Snapshot: false> Please note: The interpreters you downloaded need to be **reload**, when your Zeppelin server is down. ### UnloadIf you want to **unload** the interpreters using REST API, ```http://[zeppelin-server]:[zeppelin-port]/api/interpreter/unload/[interpreter_group_name]/[interpreter_name]```In this case, the Restful method will be **DELETE**.## What is the next step after Loading ? ### Q1. Where is the location of interpreters you downloaded ? Actually, the answer about this question is in the above picture. Once the REST API is called, the `.jar` files of interpreters you get are saved under `ZEPPELIN_HOME/local-repo` first. Then, they will be copied to `ZEPPELIN_HOME/interpreter` directory. So, please checkout your `ZEPPELIN_HOME/interpreter`.### Q2. Then, how can I use this interpreter ?After loading an interpreter, you can use it by creating and configuring it in Zeppelin's **Interpreter tab**.Oh, you don't need to restart your Zeppelin server. Because it is **Dynamic Loading**, you can configure and load it **at runtime** !1. After Zeppelin server up, browse Zeppelin home and click **Interpreter tab**.2. At the **Interpreter** section, click **+Create** button. 3. Then, you can verify the interpreter list that you loaded.4. After choosing an interpreter, you can configure and use it. Don't forget to save it.5. Create a new notebook in the **Notebook** section, then you can bind the interpreters from your interpreter list. Just drag and drop !6. At last, you can use your interpreter !If you want to get the specific information about respective interpreters, please checkout each interpreter documentation. ",
"url": " /usage/interpreter/dynamic_loading.html",
"group": "usage/interpreter",
"excerpt": "Apache Zeppelin provides pluggable interpreter architecture which results in a wide and variety of the supported backend system. In this page, we will introduce dynamic interpreter loading using REST API."
}
,
"/usage/interpreter/installation.html": {
"title": "Installing Interpreters",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Installing Interpreters Apache Zeppelin provides **Interpreter Installation** mechanism for whom downloaded Zeppelin `netinst` binary package, or just want to install another 3rd party interpreters.## Community managed interpretersApache Zeppelin provides several interpreters as [community managed interpreters](#available-community-managed-interpreters).If you downloaded `netinst` binary package, you need to install by using below commands.#### Install all community managed interpreters```bash./bin/install-interpreter.sh --all```#### Install specific interpreters```bash./bin/install-interpreter.sh --name md,shell,jdbc,python```You can get full list of community managed interpreters by running```bash./bin/install-interpreter.sh --list```#### Install interpreter built with Scala 2.10Zeppelin support both Scala 2.10 and 2.11 for several interpreters as below: Name Maven Artifact for Scala 2.10 Maven Artifact for Scala 2.11 cassandra org.apache.zeppelin:zeppelin-cassandra_2.10:0.8.0 org.apache.zeppelin:zeppelin-cassandra_2.11:0.8.0 flink org.apache.zeppelin:zeppelin-flink_2.10:0.8.0 org.apache.zeppelin:zeppelin-flink_2.11:0.8.0 ignite org.apache.zeppelin:zeppelin-ignite_2.10:0.8.0 org.apache.zeppelin:zeppelin-ignite_2.11:0.8.0 scio org.apache.zeppelin:zeppelin-scio_2.10:0.8.0 org.apache.zeppelin:zeppelin-scio_2.11:0.8.0 If you install one of these interpreters only with `--name` option, installer will download interpreter built with Scala 2.11 by default. If you want to specify Scala version, you will need to add `--artifact` option. Here is the example of installing flink interpreter built with Scala 2.10.```bash./bin/install-interpreter.sh --name flink --artifact org.apache.zeppelin:zeppelin-flink_2.10:0.8.0```#### Spark interpreter for scala 2.10 and 2.11 Starting from Zeppelin 0.8.0, Zeppelin support both scala 2.10 and 2.11 in one zeppelin instance. You don't need to install different scala specific interpreter for spark. It would pick up the right scala specific interpreter for the spark distribution where `SPARK_HOME` point to.Here's the command to install spark interpreter```bashrm -rf ./interpreter/spark./bin/install-interpreter.sh --name spark --artifact org.apache.zeppelin:spark-interpreter:0.8.0```Once you have installed interpreters, you need to restart Zeppelin. And then [create interpreter setting](./overview.html#what-is-zeppelin-interpreter) and [bind it with your notebook](./overview.html#what-is-zeppelin-interpreter-setting).## 3rd party interpretersYou can also install 3rd party interpreters located in the maven repository by using below commands.#### Install 3rd party interpreters```bash./bin/install-interpreter.sh --name interpreter1 --artifact groupId1:artifact1:version1```The above command will download maven artifact `groupId1:artifact1:version1` and all of its transitive dependencies into `interpreter/interpreter1` directory.After restart Zeppelin, then [create interpreter setting](./overview.html#what-is-zeppelin-interpreter) and [bind it with your note](./overview.html#what-is-interpreter-setting).#### Install multiple 3rd party interpreters at once```bash./bin/install-interpreter.sh --name interpreter1,interpreter2 --artifact groupId1:artifact1:version1,groupId2:artifact2:version2````--name` and `--artifact` arguments will recieve comma separated list.## Available community managed interpretersYou can also find the below community managed interpreter list in `conf/interpreter-list` file. Name Maven Artifact Description alluxio org.apache.zeppelin:zeppelin-alluxio:0.8.0 Alluxio interpreter angular org.apache.zeppelin:zeppelin-angular:0.8.0 HTML and AngularJS view rendering beam org.apache.zeppelin:zeppelin-beam:0.8.0 Beam interpreter bigquery org.apache.zeppelin:zeppelin-bigquery:0.8.0 BigQuery interpreter cassandra org.apache.zeppelin:zeppelin-cassandra_2.11:0.8.0 Cassandra interpreter built with Scala 2.11 elasticsearch org.apache.zeppelin:zeppelin-elasticsearch:0.8.0 Elasticsearch interpreter file org.apache.zeppelin:zeppelin-file:0.8.0 HDFS file interpreter flink org.apache.zeppelin:zeppelin-flink_2.11:0.8.0 Flink interpreter built with Scala 2.11 hbase org.apache.zeppelin:zeppelin-hbase:0.8.0 Hbase interpreter ignite org.apache.zeppelin:zeppelin-ignite_2.11:0.8.0 Ignite interpreter built with Scala 2.11 jdbc org.apache.zeppelin:zeppelin-jdbc:0.8.0 Jdbc interpreter kylin org.apache.zeppelin:zeppelin-kylin:0.8.0 Kylin interpreter lens org.apache.zeppelin:zeppelin-lens:0.8.0 Lens interpreter livy org.apache.zeppelin:zeppelin-livy:0.8.0 Livy interpreter md org.apache.zeppelin:zeppelin-markdown:0.8.0 Markdown support pig org.apache.zeppelin:zeppelin-pig:0.8.0 Pig interpreter python org.apache.zeppelin:zeppelin-python:0.8.0 Python interpreter scio org.apache.zeppelin:zeppelin-scio_2.11:0.8.0 Scio interpreter built with Scala 2.11 shell org.apache.zeppelin:zeppelin-shell:0.8.0 Shell command sap org.apache.zeppelin:zeppelin-sap:0.8.0 sap interpreter ",
"url": " /usage/interpreter/installation.html",
"group": "usage/interpreter",
"excerpt": "Apache Zeppelin provides Interpreter Installation mechanism for whom downloaded Zeppelin netinst binary package, or just want to install another 3rd party interpreters."
}
,
"/usage/other_features/cron_scheduler.html": {
"title": "Running a Notebook on a Given Schedule Automatically",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Running a Notebook on a Given Schedule AutomaticallyApache Zeppelin provides a cron scheduler for each notebook. You can run a notebook on a given schedule automatically by setting up a cron scheduler on the notebook.## Setting up a cron scheduler on a notebookClick the clock icon on the tool bar and open a cron scheduler dialog box.There are the following items which you can input or set:### PresetYou can set a cron schedule easily by clicking each option such as `1m` and `5m`. The login user is set as a cron executing user automatically. You can also clear the cron schedule settings by clicking `None`.### Cron expressionYou can set the cron schedule by filling in this form. Please see [Cron Trigger Tutorial](http://www.quartz-scheduler.org/documentation/quartz-2.2.x/tutorials/crontrigger) for the available cron syntax.### Cron executing user (It is removed from 0.8 where it enforces the cron execution user to be the note owner for security purpose)You can set the cron executing user by filling in this form and press the enter key.### After execution stop the interpreterWhen this checkbox is set to "on", the interpreters which are binded to the notebook are stopped automatically after the cron execution. This feature is useful if you want to release the interpreter resources after the cron execution.> **Note**: A cron execution is skipped if one of the paragraphs is in a state of `RUNNING` or `PENDING` no matter whether it is executed automatically (i.e. by the cron scheduler) or manually by a user opening this notebook.### Enable cronSet property **zeppelin.notebook.cron.enable** to **true** in `$ZEPPELIN_HOME/conf/zeppelin-site.xml` to enable Cron feature.### Run cron selectively on foldersIn `$ZEPPELIN_HOME/conf/zeppelin-site.xml` make sure the property **zeppelin.notebook.cron.enable** is set to **true**, and then set property **zeppelin.notebook.cron.folders** to the desired folder as comma-separated values, e.g. `*yst*, Sys?em, System`. This property accepts wildcard and joker.",
"url": " /usage/other_features/cron_scheduler.html",
"group": "usage/other_features",
"excerpt": "You can run a notebook on a given schedule automatically by setting up a cron scheduler on the notebook."
}
,
"/usage/other_features/zeppelin_context.html": {
"title": "Zeppelin-Context",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Zeppelin-ContextThe zeppelin-context is a system-wide container for common utility functions anduser-specific data. It implements functions for data input, data display, etc. that areoften needed but are not uniformly available in all interpreters.Its single per-user instance is accessible across all of the user's notebooks and cells,enabling data exchange between cells - even in different notebooks.But the way in which the zeppelin-context is used, and the functionality available differsdepending on whether or not the associated interpreter is based on a programming language.Details of how the zeppelin-context is used for different purposes and in differentenvironments is described below.## Usage in Programming Language CellsIn many programming-language interpreters (e.g. Apache Spark, Python, R) the zeppelin-context is availableas a predefined variable `z` that can be used by directly invoking its methods.The methods available on the `z` object are described below.Other interpreters based on programming languages like spark.dep, Apache Beam, etc. also provide thepredefined variable `z`.### Exploring Spark DataFramesIn the Apache Spark interpreter, the zeppelin-context provides a `show` method, which, using Zeppelin's `table` feature, can be used to nicely display a Spark DataFrame:```scaladf = spark.read.csv('/path/to/csv')z.show(df)```This display functionality using the `show` method is planned to be extended uniformly to other interpreters that can access the `z` object.### Object Exchange`ZeppelinContext` extends map and it's shared between the Apache Spark and Python environments.So you can put some objects using Scala (in an Apache Spark cell) and read it from Python, and vice versa. {% highlight scala %}// Put object from scala%sparkval myObject = ...z.put("objName", myObject)// Exchanging data framesmyScalaDataFrame = ...z.put("myScalaDataFrame", myScalaDataFrame)val myPythonDataFrame = z.get("myPythonDataFrame").asInstanceOf[DataFrame]{% endhighlight %} {% highlight python %}# Get object from python%spark.pysparkmyObject = z.get("objName")# Exchanging data framesmyPythonDataFrame = ...z.put("myPythonDataFrame", postsDf._jdf)myScalaDataFrame = DataFrame(z.get("myScalaDataFrame"), sqlContext){% endhighlight %} ### Form Creation`ZeppelinContext` provides functions for creating forms.In Scala and Python environments, you can create forms programmatically. {% highlight scala %}%spark/* Create text input form */z.input("formName")/* Create text input form with default value */z.input("formName", "defaultValue")/* Create select form */z.select("formName", Seq(("option1", "option1DisplayName"), ("option2", "option2DisplayName")))/* Create select form with default value*/z.select("formName", "option1", Seq(("option1", "option1DisplayName"), ("option2", "option2DisplayName"))){% endhighlight %} {% highlight python %}%spark.pyspark# Create text input formz.input("formName")# Create text input form with default valuez.input("formName", "defaultValue")# Create select formz.select("formName", [("option1", "option1DisplayName"), ("option2", "option2DisplayName")])# Create select form with default valuez.select("formName", [("option1", "option1DisplayName"), ("option2", "option2DisplayName")], "option1"){% endhighlight %} In sql environment, you can create form in simple template.```sql%spark.sqlselect * from ${table=defaultTableName} where text like '%${search}%'```To learn more about dynamic form, checkout [Dynamic Form](../usage/dynamic_form/intro.html).### Interpreter-Specific FunctionsSome interpreters use a subclass of `BaseZepplinContext` augmented with interpreter-specific functions. For example functions of the dependency loader (%spark.dep) can be invoked as `z.addRepo()`, `z.load()`, etc. Such interpreter-specific functions are described within each interpreter's documentation.## Usage with Embedded CommandsIn certain interpreters (see table below) zeppelin-context features may be invoked by embedding command strings into the paragraph text. Such embedded command strings are used to invoke dynamic-forms and object-interpolation as described below. | Interpreters that use Embedded Commands ||-------------------------------------------------------------------||spark.sql (*), bigquery, cassandra, elasticsearch, file, hbase, ignite, jdbc (*), kylin, livy, markdown, neo4j, pig, python, shell (*), zengine |Dynamic forms are available in all of the interpreters in the table above, but object interpolation is only available in a small, but growing, list of interpreters (marked with an asterisk in the table above).Both these zeppelin-context features are described below.### Dynamic FormsPatterns of the form ${ ... } are used to dynamically create additional HTML elementsfor requesting user input (that replaces the corresponding pattern in the paragraph text). Currently only [text](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/text), [select](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/select) with [options](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/option), and [checkbox](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/checkbox) are supported.Dynamic forms are described in detail here: [Dynamic Form](../usage/dynamic_form/intro.html).### Object InterpolationSome interpreters can interpolate object values from `z` into the paragraph text by using the`{variable-name}` syntax. The value of any object previously `put` into `z` can beinterpolated into a paragraph text by using such a pattern containing the object's name.The following example shows one use of this facility:####In Scala cell:```scalaz.put("minAge", 35)```####In later SQL cell:```sql%sql select * from members where age >= {minAge}```The interpolation of a `{var-name}` pattern is performed only when `z` contains an object with the specified name.But the pattern is left unchanged if the named object does not exist in `z`.Further, all `{var-name}` patterns within the paragraph text must must be translatable for any interpolation to occur --translation of only some of the patterns in a paragraph text is never done.In some situations, it is necessary to use { and } characters in a paragraph text without invoking theobject interpolation mechanism. For these cases an escaping mechanism is available --doubled braces {{ and }} should be used. The following example shows the use of {{ and }} for passing aregular expression containing just { and } into the paragraph text.```sql%sql select * from members where name rlike '[aeiou]{{3}}'```To summarize, patterns of the form `{var-name}` within the paragraph text will be interpolated only if a predefinedobject of the specified name exists. Additionally, all such patterns within the paragraph text should alsobe translatable for any interpolation to occur. Patterns of the form `{{any-text}}` are translated into `{any-text}`.These translations are performed only when all occurrences of `{`, `}`, `{{`, and `}}` in the paragraph text conformto one of the two forms described above. Paragraph text containing `{` and/or `}` characters used in any other way(than `{var-name}` and `{{any-text}}`) is used as-is without any changes.No error is flagged in any case. This behavior is identical to the implementation of a similar feature inJupyter's shell invocation using the `!` magic command.This feature is disabled by default, and must be explicitly turned on for each interpreter independentlyby setting the value of an interpreter-specific property to `true`.Consult the _Configuration_ section of each interpreter's documentationto find out if object interpolation is implemented, and the name of the parameter that must be set to `true` toenable the feature. The name of the parameter used to enable this feature it is different for each interpreter.For example, the SparkSQL and Shell interpreters use the parameter names `zeppelin.spark.sql.interpolation` and`zeppelin.shell.interpolation` respectively.At present only the SparkSQL, JDBC, and Shell interpreters support object interpolation.",
"url": " /usage/other_features/zeppelin_context.html",
"group": "usage/other_features",
"excerpt": "The Zeppelin-Context is a system-wide container for a variety of user-specific settings and parameters that are accessible across notebooks, cells, and interpreters."
}
,
"/usage/other_features/publishing_paragraphs.html": {
"title": "How can you publish your paragraphs",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# How can you publish your paragraphs?Apache Zeppelin provides a feature for publishing your notebook paragraph results. Using this feature, you can show Zeppelin notebook paragraph results in your own website. It's very straightforward. Just use `` tag in your page.## Copy a Paragraph LinkA first step to publish your paragraph result is **Copy a Paragraph Link**. * After running a paragraph in your Zeppelin notebook, click a gear button located on the right side. Then, click **Link this Paragraph** menu like below image. * Just copy the provided link. ## Embed the Paragraph to Your WebsiteFor publishing the copied paragraph, you may use `` tag in your website page.For example,```:/#/notebook/2B3QSZTKR/paragraph/...?asIframe" height="" width="" >```Finally, you can show off your beautiful visualization results in your website. > **Note**: To embed the paragraph in a website, Apache Zeppelin needs to be reachable by that website. And please use this feature with caution and in a trusted environment only, as Zeppelin entire Webapp could be accessible by whoever visits your website. ",
"url": " /usage/other_features/publishing_paragraphs.html",
"group": "usage/other_features",
"excerpt": "Apache Zeppelin provides a feature for publishing your notebook paragraph results. Using this feature, you can show Zeppelin notebook paragraph results in your own website."
}
,
"/usage/other_features/personalized_mode.html": {
"title": "Personalized Mode",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# What is Personalized Mode? ",
"url": " /usage/other_features/personalized_mode.html",
"group": "usage/other_features",
"excerpt": ""
}
,
"/usage/other_features/customizing_homepage.html": {
"title": "Customizing Apache Zeppelin homepage",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Customizing Apache Zeppelin homepageApache Zeppelin allows you to use one of the notes you create as your Zeppelin Homepage.With that you can brand your Zeppelin installation, adjust the instruction to your users needs and even translate to other languages.## How to set a note as your Zeppelin homepageThe process for creating your homepage is very simple as shown below:1. Create a note using Zeppelin2. Set the note id in the config file3. Restart Zeppelin### Create a note using ZeppelinCreate a new note using Zeppelin,you can use ```%md``` interpreter for markdown content or any other interpreter you like.You can also use the display system to generate [text](../display_system/basic.html#text), [html](../display_system/basic.html#html), [table](../display_system/basic.html#table) orAngular ([backend API](../display_system/angular_backend.html), [frontend API](../display_system/angular_frontend.html)).Run (shift+Enter) the note and see the output. Optionally, change the note view to report to hidethe code sections.### Set the note id in the config fileTo set the note id in the config file, you should copy it from the last word in the note url.For example,Set the note id to the ```ZEPPELIN_NOTEBOOK_HOMESCREEN``` environment variableor ```zeppelin.notebook.homescreen``` property.You can also set the ```ZEPPELIN_NOTEBOOK_HOMESCREEN_HIDE``` environment variableor ```zeppelin.notebook.homescreen.hide``` property to hide the new note from the note list.### Restart ZeppelinRestart your Zeppelin server```bash./bin/zeppelin-daemon stop./bin/zeppelin-daemon start```That's it! Open your browser and navigate to Apache Zeppelin and see your customized homepage.## Show note list in your custom homepageIf you want to display the list of notes on your custom Apache Zeppelin homepage allyou need to do is use our %angular support.Add the following code to a paragraph in your Apache Zeppelin note and run it.```javascript%sparkprintln("""%angular """)```After running the paragraph, you will see output similar to this one:That's it! Voila! You have your note list.",
"url": " /usage/other_features/customizing_homepage.html",
"group": "usage/other_features",
"excerpt": "Apache Zeppelin allows you to use one of the notes you create as your Zeppelin Homepage. With that you can brand your Zeppelin installation, adjust the instruction to your users needs and even translate to other languages."
}
,
"/usage/other_features/notebook_actions.html": {
"title": "Notebook Actions",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Revisions comparatorApache Zeppelin allows you to compare revisions of notebook.To see which paragraphs have been changed, removed or added.This action becomes available if your notebook has more than one revision.## How to compare two revisionsFor compare two revisions need open dialog of comparator (by click button) and click on any revision in the table.Or choose two revisions into comboboxes.After click on any revision in the table or selecting the second revision will see the result of the comparison.## How to read the result of the comparisonResult it is list of paragraphs which was in both revisions. If paragraph was added in second revision ("Head")then so it will be marked as added, if was deleted then it will be marked asdeleted. If paragraph exists in both revisions then it marked as there are differences.To view the comparison click on the section.Сhanges in the text of the paragraph are highlighted in green and red. Red it is line (block of lines) which was deleted, green it is line (block of lines) which was added).",
"url": " /usage/other_features/notebook_actions.html",
"group": "usage/other_features",
"excerpt": "Description of some actions for notebooks"
}
,
"/usage/dynamic_form/intro.html": {
"title": "Dynamic Form in Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# What is Dynamic Form?Apache Zeppelin dynamically creates input forms. Depending on language backend, there're two different ways to create dynamic form.Custom language backend can select which type of form creation it wants to use. Forms can have different scope (paragraph or note). Forms with scope "note" available in all paragraphs regardless of which paragraph has code to create these forms.## Using form Templates (scope: paragraph)This mode creates form using simple template language. It's simple and easy to use. For example Markdown, Shell, Spark SQL language backend uses it.### Text input formTo create text input form, use `${formName}` templates.for exampleAlso you can provide default value, using `${formName=defaultValue}`.### Select formTo create select form, use `${formName=defaultValue,option1|option2...}`for exampleAlso you can separate option's display name and value, using `${formName=defaultValue,option1(DisplayName)|option2(DisplayName)...}`The paragraph will be automatically run after you change your selection by default.But in case you have multiple types dynamic form in one paragraph, you might want to run the paragraph after changing all the selections.You can control this by unchecking the below **Run on selection change** option in the setting menu.Even if you uncheck this option, still you can run it by pressing `Enter`.### Checkbox formFor multi-selection, you can create a checkbox form using `${checkbox:formName=defaultValue1|defaultValue2...,option1|option2...}`. The variable will be substituted by a comma-separated string based on the selected items. For example:You can specify the delimiter using `${checkbox(delimiter):formName=...}`:Like [select form](#select-form), the paragraph will be automatically run after you change your selection by default.But in case you have multiple types dynamic form in one paragraph, you might want to run the paragraph after changing all the selections.You can control this by unchecking the below **Run on selection change** option in the setting menu.Even if you uncheck this option, still you can run it by pressing `Enter`.## Using form Templates (scope: note)Has a same syntax but starts with two symbols `$`. (for ex. input `$${forName}`)## Creates Programmatically (scope: paragraph)Some language backends can programmatically create forms. For example [ZeppelinContext](../../interpreter/spark.html#zeppelincontext) provides a form creation APIHere are some examples:### Text input form {% highlight scala %}%sparkprintln("Hello "+z.textbox("name")){% endhighlight %} {% highlight python %}%pysparkprint("Hello "+z.textbox("name")){% endhighlight %} ### Text input form with default value {% highlight scala %}%sparkprintln("Hello "+z.textbox("name", "sun")) {% endhighlight %} {% highlight python %}%pysparkprint("Hello "+z.textbox("name", "sun")){% endhighlight %} ### Select form {% highlight scala %}%sparkprintln("Hello "+z.select("day", Seq(("1","mon"), ("2","tue"), ("3","wed"), ("4","thurs"), ("5","fri"), ("6","sat"), ("7","sun")))){% endhighlight %} {% highlight python %}%pysparkprint("Hello "+z.select("day", [("1","mon"), ("2","tue"), ("3","wed"), ("4","thurs"), ("5","fri"), ("6","sat"), ("7","sun")])){% endhighlight %} #### Checkbox form {% highlight scala %}%sparkval options = Seq(("apple","Apple"), ("banana","Banana"), ("orange","Orange"))println("Hello "+z.checkbox("fruit", options).mkString(" and ")){% endhighlight %} {% highlight python %}%pysparkoptions = [("apple","Apple"), ("banana","Banana"), ("orange","Orange")]print("Hello "+ " and ".join(z.checkbox("fruit", options, ["apple"]))){% endhighlight %} ## Creates Programmatically (scope: note)The difference in the method names: Scope paragraph Scope note input (or textbox) noteTextbox select noteSelect checkbox noteCheckbox ",
"url": " /usage/dynamic_form/intro.html",
"group": "usage/dynamic_form",
"excerpt": "Apache Zeppelin dynamically creates input forms. Depending on language backend, there're two different ways to create dynamic form."
}
,
"/usage/display_system/basic.html": {
"title": "Basic Display System in Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Basic Display System in Apache Zeppelin## TextBy default, Apache Zeppelin prints interpreter response as a plain text using `text` display system.You can explicitly say you're using `text` display system.## HtmlWith `%html` directive, Zeppelin treats your output as HTML### Mathematical expressionsHTML display system automatically formats mathematical expression using [MathJax](https://www.mathjax.org/). You can use`( INLINE EXPRESSION )` and `$$ EXPRESSION $$` to format. For example## TableIf you have data that row separated by `n` (newline) and column separated by `t` (tab) with first row as header row, for exampleYou can simply use `%table` display system to leverage Zeppelin's built in visualization.If table contents start with `%html`, it is interpreted as an HTML.> **Note :** Display system is backend independent.## NetworkWith the `%network` directive, Zeppelin treats your output as a graph. Zeppelin can leverage the Property Graph Model.### What is the Labelled Property Graph Model?A [Property Graph](https://github.com/tinkerpop/gremlin/wiki/Defining-a-Property-Graph) is a graph that has these elements:* a set of vertices * each vertex has a unique identifier. * each vertex has a set of outgoing edges. * each vertex has a set of incoming edges. * each vertex has a collection of properties defined by a map from key to value* a set of edges * each edge has a unique identifier. * each edge has an outgoing tail vertex. * each edge has an incoming head vertex. * each edge has a label that denotes the type of relationship between its two vertices. * each edge has a collection of properties defined by a map from key to value.A [Labelled Property Graph](https://neo4j.com/developer/graph-database/#property-graph) is a Property Graph where the nodes can be tagged with **labels** representing their different roles in the graph model### What are the APIs?The new NETWORK visualization is based on json with the following params:* "nodes" (mandatory): list of nodes of the graph every node can have the following params: * "id" (mandatory): the id of the node (must be unique); * "label": the main Label of the node; * "labels": the list of the labels of the node; * "data": the data attached to the node;* "edges": list of the edges of the graph; * "id" (mandatory): the id of the edge (must be unique); * "source" (mandatory): the id of source node of the edge; * "target" (mandatory): the id of target node of the edge; * "label": the main type of the edge; * "data": the data attached to the edge;* "labels": a map (K, V) where K is the node label and V is the color of the node;* "directed": (true/false, default false) wich tells if is directed graph or not;* "types": a *distinct* list of the edge types of the graphIf you click on a node or edge on the bottom of the paragraph you find a list of entity propertiesThis kind of graph can be easily *flatten* in order to support other visualization formats provided by Zeppelin.### How to use it?An example of a simple graph```scala%sparkprint(s"""%network { "nodes": [ {"id": 1}, {"id": 2}, {"id": 3} ], "edges": [ {"source": 1, "target": 2, "id" : 1}, {"source": 2, "target": 3, "id" : 2}, {"source": 1, "target": 2, "id" : 3}, {"source": 1, "target": 2, "id" : 4}, {"source": 2, "target": 1, "id" : 5}, {"source": 2, "target": 1, "id" : 6} ]}""")```that will look like:A little more complex graph:```scala%sparkprint(s"""%network { "nodes": [{"id": 1, "label": "User", "data": {"fullName":"Andrea Santurbano"}},{"id": 2, "label": "User", "data": {"fullName":"Lee Moon Soo"}},{"id": 3, "label": "Project", "data": {"name":"Zeppelin"}}], "edges": [{"source": 2, "target": 1, "id" : 1, "label": "HELPS"},{"source": 2, "target": 3, "id" : 2, "label": "CREATE"},{"source": 1, "target": 3, "id" : 3, "label": "CONTRIBUTE_TO", "data": {"oldPR": "https://github.com/apache/zeppelin/pull/1582"}}], "labels": {"User": "#8BC34A", "Project": "#3071A9"}, "directed": true, "types": ["HELPS", "CREATE", "CONTRIBUTE_TO"]}""")```that will look like:",
"url": " /usage/display_system/basic.html",
"group": "usage/display_system",
"excerpt": "There are 3 basic display systems in Apache Zeppelin. By default, Zeppelin prints interpreter responce as a plain text using text display system. With %html directive, Zeppelin treats your output as HTML. You can also simply use %table display system..."
}
,
"/usage/display_system/angular_backend.html": {
"title": "Backend Angular API in Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Backend Angular API in Apache Zeppelin## OverviewAngular display system treats output as a view template for [AngularJS](https://angularjs.org/).It compiles templates and displays them inside of Apache Zeppelin. Zeppelin provides a gateway between your interpreter and your compiled **AngularJS view** templates.Therefore, you can not only update scope variables from your interpreter but also watch them in the interpreter, which is JVM process.## Basic Usage ### Print AngularJS viewTo use angular display system, you should start with `%angular`.Since `name` is not defined, `Hello {{name}}` will display `Hello`.> **Please Note:** Display system is backend independent.### Bind / Unbind VariablesThrough **ZeppelinContext**, you can bind / unbind variables to **AngularJS view**. Currently, it only works in **Spark Interpreter ( scala )**.```scala// bind my 'object' as angular scope variable 'name' in current notebook.z.angularBind(String name, Object object)// bind my 'object' as angular scope variable 'name' in all notebooks related to current interpreter.z.angularBindGlobal(String name, Object object)// unbind angular scope variable 'name' in current notebook.z.angularUnbind(String name)// unbind angular scope variable 'name' in all notebooks related to current interpreter.z.angularUnbindGlobal(String name)```Using the above example, let's bind `world` variable to `name`. Then you can see **AngularJs view** is immediately updated.### Watch / Unwatch VariablesThrough **ZeppelinContext**, you can watch / unwatch variables in **AngularJs view**. Currently, it only works in **Spark Interpreter ( scala )**.```scala// register for angular scope variable 'name' (notebook)z.angularWatch(String name, (before, after) => { ... })// unregister watcher for angular variable 'name' (notebook)z.angularUnwatch(String name)// register for angular scope variable 'name' (global)z.angularWatchGlobal(String name, (before, after) => { ... })// unregister watcher for angular variable 'name' (global)z.angularUnwatchGlobal(String name)```Let's make a button. When it is clicked, the value of `run` will be increased 1 by 1.`z.angularBind("run", 0)` will initialize `run` to zero. And then, it will be also applied to `run` in `z.angularWatch()`.When the button is clicked, you'll see both `run` and `numWatched` are incremented by 1.## Let's make it Simpler and more IntuitiveIn this section, we will introduce a simpler and more intuitive way of using **Angular Display System** in Zeppelin.Here are some usages.### Import```scala// In notebook scopeimport org.apache.zeppelin.display.angular.notebookscope._import AngularElem._// In paragraph scopeimport org.apache.zeppelin.display.angular.paragraphscope._import AngularElem._```### Display Element```scala// automatically convert to string and print with %angular display system directive in front..display```### Event Handler```scala// on click.onClick(() => { my callback routine}).display// on change.onChange(() => { my callback routine}).display// arbitrary event.onEvent("ng-click", () => { my callback routine}).display```### Bind Model```scala// bind model.model("myModel").display// bind model with initial value.model("myModel", initialValue).display```### Interact with Model```scala// read modelAngularModel("myModel")()// update modelAngularModel("myModel", "newValue")```### Example: Basic UsageUsing the above basic usages, you can apply them like below examples.#### Display Elements```scala Hello Angular Display System.display```#### OnClick Event```scala Click me.onClick{() => // callback for button click}.display```#### Bind Model{% raw %}```scala {{{{myModel}}}}.model("myModel", "Initial Value").display```{% endraw %}#### Interact With Model```scala// read the valueAngularModel("myModel")()// update the valueAngularModel("myModel", "New value")```### Example: String ConverterUsing below example, you can convert the lowercase string to uppercase.{% raw %}```scala// clear previously created angular object.AngularElem.disassociateval button = Convert.onClick{() => val inputString = AngularModel("input")().toString AngularModel("title", inputString.toUpperCase)} { {{{{title}}}}.model("title", "Please type text to convert uppercase") } Your text { .model("input", "") } {button}.display```{% endraw %}",
"url": " /usage/display_system/angular_backend.html",
"group": "usage/display_system",
"excerpt": "Apache Zeppelin provides a gateway between your interpreter and your compiled AngularJS view templates. You can not only update scope variables from your interpreter but also watch them in the interpreter, which is JVM process."
}
,
"/usage/display_system/angular_frontend.html": {
"title": "Frontend Angular API in Apache Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Frontend Angular API in Apache Zeppelin## Basic UsageIn addition to the [backend Angular API](./angular_backend.html) to handle Angular objects binding, Apache Zeppelin also exposes a simple AngularJS **z** object on the front-end side to expose the same capabilities.This **z** object is accessible in the Angular isolated scope for each paragraph.### Bind / Unbind VariablesThrough the **`z`**, you can bind / unbind variables to **AngularJS view**.Bind a value to an angular object and a **mandatory** target paragraph:```html%angular Super Hero: Bind```Unbind/remove a value from angular object and a **mandatory** target paragraph:```html%angular UnBind```The signature for the **`z.angularBind() / z.angularUnbind()`** functions are:```javascript// Bindz.angularBind(angularObjectName, angularObjectValue, paragraphId);// Unbindz.angularUnbind(angularObjectName, angularObjectValue, paragraphId);```All the parameters are mandatory.### Run ParagraphYou can also trigger paragraph execution by calling **`z.runParagraph()`** function passing the appropriate paragraphId: ```html%angular Paragraph Id: Run Paragraph```## Overriding dynamic form with Angular ObjectThe front-end Angular Interaction API has been designed to offer richer form capabilities and variable binding. With the existing **Dynamic Form** system you can already create input text, select and checkbox forms but the choice is rather limited and the look & feel cannot be changed.The idea is to create a custom form using plain HTML/AngularJS code and bind actions on this form to push/remove Angular variables to targeted paragraphs using this new API. Consequently if you use the **Dynamic Form** syntax in a paragraph and there is a bound Angular object having the same name as the `${formName}`, the Angular object will have higher priority and the **Dynamic Form** will not be displayed. Example: ## Feature matrix comparisonHow does the front-end AngularJS API compares to the [backend Angular API](./angular_backend.html)? Below is a comparison matrix for both APIs: Actions Front-end API Back-end API Initiate binding z.angularbind(var, initialValue, paragraphId) z.angularBind(var, initialValue) Update value same to ordinary angularjs scope variable, or z.angularbind(var, newValue, paragraphId) z.angularBind(var, newValue) Watching value same to ordinary angularjs scope variable z.angularWatch(var, (oldVal, newVal) => ...) Destroy binding z.angularUnbind(var, paragraphId) z.angularUnbind(var) Executing Paragraph z.runParagraph(paragraphId) z.run(paragraphId) Executing Paragraph (Specific paragraphs in other notes) ( z.run(noteid, paragraphId) Executing note z.runNote(noteId) Both APIs are pretty similar, except for value watching where it is done naturally by AngularJS internals on the front-end and by user custom watcher functions in the back-end.There is also a slight difference in term of scope. Front-end API limits the Angular object binding to a paragraph scope whereas back-end API allows you to bind an Angular object at the global or note scope. This restriction has been designed purposely to avoid Angular object leaks and scope pollution.",
"url": " /usage/display_system/angular_frontend.html",
"group": "usage/display_system",
"excerpt": "In addition to the back-end API to handle Angular objects binding, Apache Zeppelin exposes a simple AngularJS z object on the front-end side to expose the same capabilities."
}
,
"/usage/rest_api/zeppelin_server.html": {
"title": "Apache Zeppelin Server REST API",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Apache Zeppelin Server REST API## OverviewApache Zeppelin provides several REST APIs for interaction and remote activation of zeppelin functionality.All REST APIs are available starting with the following endpoint `http://[zeppelin-server]:[zeppelin-port]/api`. Note that Apache Zeppelin REST APIs receive or return JSON objects, it is recommended for you to install some JSON viewers such as [JSONView](https://chrome.google.com/webstore/detail/jsonview/chklaanhfefbnpoihckbnefhakgolnmc).If you work with Apache Zeppelin and find a need for an additional REST API, please [file an issue or send us an email](http://zeppelin.apache.org/community.html).## Zeppelin Server REST API list### Get Zeppelin version Description This ```GET``` method returns Zeppelin version URL ```http://[zeppelin-server]:[zeppelin-port]/api/version``` Success code 200 Fail code 500 sample JSON response { "status": "OK", "message": "Zeppelin version", "body": [ { "version": "0.8.0", "git-commit-id": "abc0123", "git-timestamp": "2017-01-02 03:04:05" } ]} ### Change the log level of Zeppelin Server Description This ```PUT``` method is used to update the root logger's log level of the server. URL ```http://[zeppelin-server]:[zeppelin-port]/api/log/level/``` Success code 200 Fail code 406 sample JSON response { "status": "OK"} sample error JSON response { "status":"NOT_ACCEPTABLE", "message":"Please check LOG level specified. Valid values: DEBUG, ERROR, FATAL, INFO, TRACE, WARN"} ",
"url": " /usage/rest_api/zeppelin_server.html",
"group": "usage/rest_api",
"excerpt": "This page contains Apache Zeppelin Server REST API information."
}
,
"/usage/rest_api/interpreter.html": {
"title": "Apache Zeppelin Interpreter REST API",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Apache Zeppelin Interpreter REST API## OverviewApache Zeppelin provides several REST APIs for interaction and remote activation of zeppelin functionality.All REST APIs are available starting with the following endpoint `http://[zeppelin-server]:[zeppelin-port]/api`. Note that Apache Zeppelin REST APIs receive or return JSON objects, it is recommended for you to install some JSON viewers such as [JSONView](https://chrome.google.com/webstore/detail/jsonview/chklaanhfefbnpoihckbnefhakgolnmc).If you work with Apache Zeppelin and find a need for an additional REST API, please [file an issue or send us an email](http://zeppelin.apache.org/community.html).## Interpreter REST API ListThe role of registered interpreters, settings and interpreters group are described in [here](../interpreter/overview.html).### List of registered interpreters Description This ```GET``` method returns all the registered interpreters available on the server. URL ```http://[zeppelin-server]:[zeppelin-port]/api/interpreter``` Success code 200 Fail code 500 Sample JSON response { "status": "OK", "message": "", "body": { "md.md": { "name": "md", "group": "md", "className": "org.apache.zeppelin.markdown.Markdown", "properties": {}, "path": "/zeppelin/interpreter/md" }, "spark.spark": { "name": "spark", "group": "spark", "className": "org.apache.zeppelin.spark.SparkInterpreter", "properties": { "spark.executor.memory": { "name": "spark.executor.memory", "defaultValue": "1g", "description": "Executor memory per worker instance. ex) 512m, 32g", "type": "string" }, "spark.cores.max": { "defaultValue": "", "description": "Total number of cores to use. Empty value uses all available core.", "type": "number" }, }, "path": "/zeppelin/interpreter/spark" }, "spark.sql": { "name": "sql", "group": "spark", "className": "org.apache.zeppelin.spark.SparkSqlInterpreter", "properties": { "zeppelin.spark.maxResult": { "name": "zeppelin.spark.maxResult", "defaultValue": "1000", "description": "Max number of Spark SQL result to display.", "type": "number" } }, "path": "/zeppelin/interpreter/spark" } }} ### List of registered interpreter settings Description This ```GET``` method returns all the interpreters settings registered on the server. URL ```http://[zeppelin-server]:[zeppelin-port]/api/interpreter/setting``` Success code 200 Fail code 500 Sample JSON response { "status": "OK", "message": "", "body": [ { "id": "2AYUGP2D5", "name": "md", "group": "md", "properties": { "_empty_": "" }, "interpreterGroup": [ { "class": "org.apache.zeppelin.markdown.Markdown", "name": "md" } ], "dependencies": [] }, { "id": "2AY6GV7Q3", "name": "spark", "group": "spark", "properties": { "spark.cores.max": { "name": "", "value": "spark.cores.max", "type": "number" }, "spark.executor.memory": { "name": "spark.executor.memory", "value": "1g", "type": "string" } }, "interpreterGroup": [ { "class": "org.apache.zeppelin.spark.SparkInterpreter", "name": "spark" }, { "class": "org.apache.zeppelin.spark.SparkSqlInterpreter", "name": "sql" } ], "dependencies": [ { "groupArtifactVersion": "com.databricks:spark-csv_2.10:1.3.0" } ] } ]} ### Get a registered interpreter setting by the setting id Description This ```GET``` method returns a registered interpreter setting on the server. URL ```http://[zeppelin-server]:[zeppelin-port]/api/interpreter/setting/[setting ID]``` Success code 200 Fail code 400 if such interpreter setting id does not exist 500 for any other errors Sample JSON response { "status": "OK", "message": "", "body": { "id": "2AYW25ANY", "name": "Markdown setting name", "group": "md", "properties": { "propname": { "name": "propname", "value": "propvalue", "type": "textarea" } }, "interpreterGroup": [ { "class": "org.apache.zeppelin.markdown.Markdown", "name": "md" } ], "dependencies": [ { "groupArtifactVersion": "groupId:artifactId:version", "exclusions": [ "groupId:artifactId" ] } ] }} ### Create a new interpreter setting Description This ```POST``` method adds a new interpreter setting using a registered interpreter to the server. URL ```http://[zeppelin-server]:[zeppelin-port]/api/interpreter/setting``` Success code 200 Fail code 400 if the input json is empty 500 for any other errors Sample JSON input { "name": "Markdown setting name", "group": "md", "properties": { "propname": { "name": "propname", "value": "propvalue", "type": "textarea" }, "interpreterGroup": [ { "class": "org.apache.zeppelin.markdown.Markdown", "name": "md" } ], "dependencies": [ { "groupArtifactVersion": "groupId:artifactId:version", "exclusions": [ "groupId:artifactId" ] } ]} Sample JSON response { "status": "OK", "message": "", "body": { "id": "2AYW25ANY", "name": "Markdown setting name", "group": "md", "properties": { "propname": { "name": "propname", "value": "propvalue", "type": "textarea" }, "interpreterGroup": [ { "class": "org.apache.zeppelin.markdown.Markdown", "name": "md" } ], "dependencies": [ { "groupArtifactVersion": "groupId:artifactId:version", "exclusions": [ "groupId:artifactId" ] } ] }} ### Update an interpreter setting Description This ```PUT``` method updates an interpreter setting with new properties. URL ```http://[zeppelin-server]:[zeppelin-port]/api/interpreter/setting/[interpreter ID]``` Success code 200 Fail code 500 Sample JSON input { "name": "Markdown setting name", "group": "md", "properties": { "propname": { "name": "propname", "value": "Otherpropvalue", "type": "textarea" }, "interpreterGroup": [ { "class": "org.apache.zeppelin.markdown.Markdown", "name": "md" } ], "dependencies": [ { "groupArtifactVersion": "groupId:artifactId:version", "exclusions": [ "groupId:artifactId" ] } ]} Sample JSON response { "status": "OK", "message": "", "body": { "id": "2AYW25ANY", "name": "Markdown setting name", "group": "md", "properties": { "propname": { "name": "propname", "value": "Otherpropvalue", "type": "textarea" }, "interpreterGroup": [ { "class": "org.apache.zeppelin.markdown.Markdown", "name": "md" } ], "dependencies": [ { "groupArtifactVersion": "groupId:artifactId:version", "exclusions": [ "groupId:artifactId" ] } ] }} ### Delete an interpreter setting Description This ```DELETE``` method deletes an given interpreter setting. URL ```http://[zeppelin-server]:[zeppelin-port]/api/interpreter/setting/[interpreter ID]``` Success code 200 Fail code 500 Sample JSON response {"status":"OK"} ### Restart an interpreter Description This ```PUT``` method restarts the given interpreter id. URL ```http://[zeppelin-server]:[zeppelin-port]/api/interpreter/setting/restart/[interpreter ID]``` Success code 200 Fail code 500 Sample JSON input (Optional) { "noteId": "2AVQJVC8N"} Sample JSON response {"status":"OK"} ### Add a new repository for dependency resolving Description This ```POST``` method adds new repository. URL ```http://[zeppelin-server]:[zeppelin-port]/api/interpreter/repository``` Success code 200 Fail code 500 Sample JSON input { "id": "securecentral", "url": "https://repo1.maven.org/maven2", "snapshot": false} Sample JSON response {"status":"OK"} ### Delete a repository for dependency resolving Description This ```DELETE``` method delete repository with given id. URL ```http://[zeppelin-server]:[zeppelin-port]/api/interpreter/repository/[repository ID]``` Success code 200 Fail code 500 ### Get available types for property Description This ```GET``` method returns available types for interpreter property. URL ```http://[zeppelin-server]:[zeppelin-port]/api/interpreter/property/types``` Success code 200 Fail code 500 Sample JSON response { "status": "OK", "body": [ "textarea", "string", ... ]} ### Get interpreter settings metadata info Description This ```GET``` method returns interpreter settings metadata info. URL ```http://[zeppelin-server]:[zeppelin-port]/api/interpreter/metadata/[setting ID]``` Success code 200 Fail code 500 ",
"url": " /usage/rest_api/interpreter.html",
"group": "usage/rest_api",
"excerpt": "This page contains Apache Zeppelin Interpreter REST API information."
}
,
"/usage/rest_api/credential.html": {
"title": "Apache Zeppelin Credential REST API",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Apache Zeppelin Credential REST API## OverviewApache Zeppelin provides several REST APIs for interaction and remote activation of zeppelin functionality.All REST APIs are available starting with the following endpoint `http://[zeppelin-server]:[zeppelin-port]/api`. Note that Apache Zeppelin REST APIs receive or return JSON objects, it is recommended for you to install some JSON viewers such as [JSONView](https://chrome.google.com/webstore/detail/jsonview/chklaanhfefbnpoihckbnefhakgolnmc).If you work with Apache Zeppelin and find a need for an additional REST API, please [file an issue or send us an email](http://zeppelin.apache.org/community.html).## Credential REST API List### List Credential information Description This ```GET``` method returns all key/value pairs of the credential information on the server. URL ```http://[zeppelin-server]:[zeppelin-port]/api/credential``` Success code 200 Fail code 500 sample JSON response { "status": "OK", "message": "", "body": { "userCredentials":{ "entity1":{ "username":"user1", "password":"password1" }, "entity2":{ "username":"user2", "password":"password2" } } }} ### Create an Credential Information Description This ```PUT``` method creates the credential information with new properties. URL ```http://[zeppelin-server]:[zeppelin-port]/api/credential/``` Success code 200 Fail code 500 Sample JSON input { "entity": "e1", "username": "user", "password": "password"} Sample JSON response { "status": "OK"} ### Delete all Credential Information Description This ```DELETE``` method deletes the credential information. URL ```http://[zeppelin-server]:[zeppelin-port]/api/credential``` Success code 200 Fail code 500 Sample JSON response {"status":"OK"} ### Delete an Credential entity Description This ```DELETE``` method deletes a given credential entity. URL ```http://[zeppelin-server]:[zeppelin-port]/api/credential/[entity]``` Success code 200 Fail code 500 Sample JSON response {"status":"OK"} ",
"url": " /usage/rest_api/credential.html",
"group": "usage/rest_api",
"excerpt": "This page contains Apache Zeppelin Credential REST API information."
}
,
"/usage/rest_api/notebook.html": {
"title": "Apache Zeppelin Notebook REST API",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Apache Zeppelin Notebook REST API## OverviewApache Zeppelin provides several REST APIs for interaction and remote activation of zeppelin functionality.All REST APIs are available starting with the following endpoint `http://[zeppelin-server]:[zeppelin-port]/api`. Note that Apache Zeppelin REST APIs receive or return JSON objects, it is recommended for you to install some JSON viewers such as [JSONView](https://chrome.google.com/webstore/detail/jsonview/chklaanhfefbnpoihckbnefhakgolnmc).If you work with Apache Zeppelin and find a need for an additional REST API, please [file an issue or send us an email](http://zeppelin.apache.org/community.html).Notebooks REST API supports the following operations: List, Create, Get, Delete, Clone, Run, Export, Import as detailed in the following tables.## Note operations### List of the notes Description This ```GET``` method lists the available notes on your server. Notebook JSON contains the ```name``` and ```id``` of all notes. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook``` Success code 200 Fail code 500 sample JSON response { "status": "OK", "message": "", "body": [ { "name":"Homepage", "id":"2AV4WUEMK" }, { "name":"Zeppelin Tutorial", "id":"2A94M5J1Z" } ]} ### Create a new note Description This ```POST``` method creates a new note using the given name or default name if none given. The body field of the returned JSON contains the new note id. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook``` Success code 200 Fail code 500 sample JSON input (without paragraphs) {"name": "name of new note"} sample JSON input (with initial paragraphs) { "name": "name of new note", "paragraphs": [ { "title": "paragraph title1", "text": "paragraph text1" }, { "title": "paragraph title2", "text": "paragraph text2", "config": { "title": true, "colWidth": 6.0, "results": [ { "graph": { "mode": "scatterChart", "optionOpen": true } } ] } } ]} sample JSON response { "status": "OK", "message": "", "body": "2AZPHY918"} ### Get the status of all paragraphs Description This ```GET``` method gets the status of all paragraphs by the given note id. The body field of the returned JSON contains of the array that compose of the paragraph id, paragraph status, paragraph finish date, paragraph started date. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/job/[noteId]``` Success code 200 Fail code 500 sample JSON response { "status": "OK", "body": [ { "id":"20151121-212654_766735423", "status":"FINISHED", "finished":"Tue Nov 24 14:21:40 KST 2015", "started":"Tue Nov 24 14:21:39 KST 2015" }, { "progress":"1", "id":"20151121-212657_730976687", "status":"RUNNING", "finished":"Tue Nov 24 14:21:35 KST 2015", "started":"Tue Nov 24 14:21:40 KST 2015" } ]} ### Get an existing note information Description This ```GET``` method retrieves an existing note's information using the given id. The body field of the returned JSON contain information about paragraphs in the note. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]``` Success code 200 Fail code 500 sample JSON response { "status": "OK", "message": "", "body": { "paragraphs": [ { "text": "%sql nselect age, count(1) valuenfrom bank nwhere age < 30 ngroup by age norder by age", "config": { "colWidth": 4, "graph": { "mode": "multiBarChart", "height": 300, "optionOpen": false, "keys": [ { "name": "age", "index": 0, "aggr": "sum" } ], "values": [ { "name": "value", "index": 1, "aggr": "sum" } ], "groups": [], "scatter": { "xAxis": { "name": "age", "index": 0, "aggr": "sum" }, "yAxis": { "name": "value", "index": 1, "aggr": "sum" } } } }, "settings": { "params": {}, "forms": {} }, "jobName": "paragraph_1423500782552_-1439281894", "id": "20150210-015302_1492795503", "results": { "code": "SUCCESS", "msg": [ { "type": "TABLE", "data": "agetvaluen19t4n20t3n21t7n22t9n23t20n24t24n25t44n26t77n27t94n28t103n29t97n" } ] }, "dateCreated": "Feb 10, 2015 1:53:02 AM", "dateStarted": "Jul 3, 2015 1:43:17 PM", "dateFinished": "Jul 3, 2015 1:43:23 PM", "status": "FINISHED", "progressUpdateIntervalMs": 500 } ], "name": "Zeppelin Tutorial", "id": "2A94M5J1Z", "angularObjects": {}, "config": { "looknfeel": "default" }, "info": {} }} ### Delete a note Description This ```DELETE``` method deletes a note by the given note id. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]``` Success code 200 Fail code 500 sample JSON response {"status": "OK","message": ""} ### Clone a note Description This ```POST``` method clones a note by the given id and create a new note using the given name or default name if none given. The body field of the returned JSON contains the new note id. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]``` Success code 200 Fail code 500 sample JSON input {"name": "name of new note"} sample JSON response { "status": "OK", "message": "", "body": "2AZPHY918"} ### Export a note Description This ```GET``` method exports a note by the given id and gernerates a JSON URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/export/[noteId]``` Success code 201 Fail code 500 sample JSON response { "paragraphs": [ { "text": "%md This is my new paragraph in my new note", "dateUpdated": "Jan 8, 2016 4:49:38 PM", "config": { "enabled": true }, "settings": { "params": {}, "forms": {} }, "jobName": "paragraph_1452300578795_1196072540", "id": "20160108-164938_1685162144", "dateCreated": "Jan 8, 2016 4:49:38 PM", "status": "READY", "progressUpdateIntervalMs": 500 } ], "name": "source note for export", "id": "2B82H3RR1", "angularObjects": {}, "config": {}, "info": {}} ### Import a note Description This ```POST``` method imports a note from the note JSON input URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/import``` Success code 201 Fail code 500 sample JSON input { "paragraphs": [ { "text": "%md This is my new paragraph in my new note", "dateUpdated": "Jan 8, 2016 4:49:38 PM", "config": { "enabled": true }, "settings": { "params": {}, "forms": {} }, "jobName": "paragraph_1452300578795_1196072540", "id": "20160108-164938_1685162144", "dateCreated": "Jan 8, 2016 4:49:38 PM", "status": "READY", "progressUpdateIntervalMs": 500 } ], "name": "source note for export", "id": "2B82H3RR1", "angularObjects": {}, "config": {}, "info": {}} sample JSON response { "status": "OK", "message": "", "body": "2AZPHY918"} ### Run all paragraphs Description This ```POST``` method runs all paragraphs in the given note id. If you can not find Note id 404 returns. If there is a problem with the interpreter returns a 412 error. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/job/[noteId]``` Success code 200 Fail code 404 or 412 sample JSON response {"status": "OK"} sample JSON error response { "status": "NOT_FOUND", "message": "note not found." } { "status": "PRECONDITION_FAILED", "message": "paragraph_1469771130099_-278315611 Not selected or Invalid Interpreter bind" } ### Stop all paragraphs Description This ```DELETE``` method stops all paragraphs in the given note id. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/job/[noteId]``` Success code 200 Fail code 500 sample JSON response {"status":"OK"} ### Clear all paragraph result Description This ```PUT``` method clear all paragraph results from note of given id. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/clear``` Success code 200 Forbidden code 401 Not Found code 404 Fail code 500 sample JSON response {"status": "OK"} ## Paragraph operations ### Create a new paragraph Description This ```POST``` method create a new paragraph using JSON payload. The body field of the returned JSON contain the new paragraph id. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/paragraph``` Success code 201 Fail code 500 sample JSON input (add to the last) { "title": "Paragraph insert revised", "text": "%sparknprintln("Paragraph insert revised")"} sample JSON input (add to specific index) { "title": "Paragraph insert revised", "text": "%sparknprintln("Paragraph insert revised")", "index": 0} sample JSON input (providing paragraph config) { "title": "paragraph title2", "text": "paragraph text2", "config": { "title": true, "colWidth": 6.0, "results": [ { "graph": { "mode": "pieChart", "optionOpen": true } } ] }} sample JSON response { "status": "OK", "message": "", "body": "20151218-100330_1754029574"} ### Get a paragraph information Description This ```GET``` method retrieves an existing paragraph's information using the given id. The body field of the returned JSON contain information about paragraph. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/paragraph/[paragraphId]``` Success code 200 Fail code 500 sample JSON response { "status": "OK", "message": "", "body": { "title": "Paragraph2", "text": "%sparknnprintln("it's paragraph2")", "dateUpdated": "Dec 18, 2015 7:33:54 AM", "config": { "colWidth": 12, "graph": { "mode": "table", "height": 300, "optionOpen": false, "keys": [], "values": [], "groups": [], "scatter": {} }, "enabled": true, "title": true, "editorMode": "ace/mode/scala" }, "settings": { "params": {}, "forms": {} }, "jobName": "paragraph_1450391574392_-1890856722", "id": "20151218-073254_1105602047", "results": { "code": "SUCCESS", "msg": [ { "type": "TEXT", "data": "it's paragraph2n" } ] }, "dateCreated": "Dec 18, 2015 7:32:54 AM", "dateStarted": "Dec 18, 2015 7:33:55 AM", "dateFinished": "Dec 18, 2015 7:33:55 AM", "status": "FINISHED", "progressUpdateIntervalMs": 500 }} ### Get the status of a single paragraph Description This ```GET``` method gets the status of a single paragraph by the given note and paragraph id. The body field of the returned JSON contains of the array that compose of the paragraph id, paragraph status, paragraph finish date, paragraph started date. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/job/[noteId]/[paragraphId]``` Success code 200 Fail code 500 sample JSON response { "status": "OK", "body": { "id":"20151121-212654_766735423", "status":"FINISHED", "finished":"Tue Nov 24 14:21:40 KST 2015", "started":"Tue Nov 24 14:21:39 KST 2015" }} ### Update paragraph Description This ```PUT``` method update paragraph contents using given id, e.g. {"text": "hello"} URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/paragraph/[paragraphId]``` Success code 200 Bad Request code 400 Forbidden code 403 Not Found code 404 Fail code 500 sample JSON input { "title": "Hello world", "text": "println("hello world")"} sample JSON response { "status": "OK", "message": "" }} ### Update paragraph configuration Description This ```PUT``` method update paragraph configuration using given id so that user can change paragraph setting such as graph type, show or hide editor/result and paragraph size, etc. You can update certain fields you want, for example you can update colWidth field only by sending request with payload {"colWidth": 12.0}. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/paragraph/[paragraphId]/config``` Success code 200 Bad Request code 400 Forbidden code 403 Not Found code 404 Fail code 500 sample JSON input { "colWidth": 6.0, "graph": { "mode": "lineChart", "height": 200.0, "optionOpen": false, "keys": [ { "name": "age", "index": 0.0, "aggr": "sum" } ], "values": [ { "name": "value", "index": 1.0, "aggr": "sum" } ], "groups": [], "scatter": {} }, "editorHide": true, "editorMode": "ace/mode/markdown", "tableHide": false} sample JSON response { "status":"OK", "message":"", "body":{ "text":"%sql nselect age, count(1) valuenfrom bank nwhere age u003c 30 ngroup by age norder by age", "config":{ "colWidth":6.0, "graph":{ "mode":"lineChart", "height":200.0, "optionOpen":false, "keys":[ { "name":"age", "index":0.0, "aggr":"sum" } ], "values":[ { "name":"value", "index":1.0, "aggr":"sum" } ], "groups":[], "scatter":{} }, "tableHide":false, "editorMode":"ace/mode/markdown", "editorHide":true }, "settings":{ "params":{}, "forms":{} }, "apps":[], "jobName":"paragraph_1423500782552_-1439281894", "id":"20150210-015302_1492795503", "results":{ "code":"SUCCESS", "msg": [ { "type":"TABLE", "data":"agetvaluen19t4n20t3n21t7n22t9n23t20n24t24n25t44n26t77n27t94n28t103n29t97n" } ] }, "dateCreated":"Feb 10, 2015 1:53:02 AM", "dateStarted":"Jul 3, 2015 1:43:17 PM", "dateFinished":"Jul 3, 2015 1:43:23 PM", "status":"FINISHED", "progressUpdateIntervalMs":500 }} ### Delete a paragraph Description This ```DELETE``` method deletes a paragraph by the given note and paragraph id. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/paragraph/[paragraphId]``` Success code 200 Fail code 500 sample JSON response {"status": "OK","message": ""} ### Run a paragraph asynchronously Description This ```POST``` method runs the paragraph asynchronously by given note and paragraph id. This API always return SUCCESS even if the execution of the paragraph fails later because the API is asynchronous URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/job/[noteId]/[paragraphId]``` Success code 200 Fail code 500 sample JSON input (optional, only needed when if you want to update dynamic form's value) { "name": "name of new note", "params": { "formLabel1": "value1", "formLabel2": "value2" }} sample JSON response {"status": "OK"} ### Run a paragraph synchronously Description This ```POST``` method runs the paragraph synchronously by given note and paragraph id. This API can return SUCCESS or ERROR depending on the outcome of the paragraph execution URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/run/[noteId]/[paragraphId]``` Success code 200 Fail code 500 sample JSON input (optional, only needed when if you want to update dynamic form's value) { "name": "name of new note", "params": { "formLabel1": "value1", "formLabel2": "value2" }} sample JSON response {"status": "OK"} sample JSON error { "status": "INTERNAL_SERVER_ERROR", "body": { "code": "ERROR", "type": "TEXT", "msg": "bash: -c: line 0: unexpected EOF while looking for matching ``'nbash: -c: line 1: syntax error: unexpected end of filenExitValue: 2" }} ### Stop a paragraph Description This ```DELETE``` method stops the paragraph by given note and paragraph id. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/job/[noteId]/[paragraphId]``` Success code 200 Fail code 500 sample JSON response {"status": "OK"} ### Move a paragraph to the specific index Description This ```POST``` method moves a paragraph to the specific index (order) from the note. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/paragraph/[paragraphId]/move/[newIndex]``` Success code 200 Fail code 500 sample JSON response {"status": "OK","message": ""} ### Full text search through the paragraphs in all notes Description ```GET``` request will return list of matching paragraphs URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/search?q=[query]``` Success code 200 Fail code 500 Sample JSON response { "status": "OK", "body": [ { "id": "/paragraph/", "name":"Note Name", "snippet":"", "text":"" } ]} ## Cron jobs### Add Cron Job Description This ```POST``` method adds cron job by the given note id. Default value of ```releaseResource``` is ```false```. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/cron/[noteId]``` Success code 200 Fail code 500 sample JSON input {"cron": "cron expression of note", "releaseResource": "false"} sample JSON response {"status": "OK"} ### Remove Cron Job Description This ```DELETE``` method removes cron job by the given note id. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/cron/[noteId]``` Success code 200 Fail code 500 sample JSON response {"status": "OK"} ### Get Cron Job Description This ```GET``` method gets cron job expression of given note id. The body field of the returned JSON contains the cron expression and ```releaseResource``` flag. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/cron/[noteId]``` Success code 200 Fail code 500 sample JSON response { "status": "OK", "body": { "cron": "0 0/1 * * * ?", "releaseResource": true }} ## Permission### Get a note permission information Description This ```GET``` method gets a note authorization information. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/permissions``` Success code 200 Forbidden code 403 Fail code 500 sample JSON response { "status":"OK", "message":"", "body":{ "readers":[ "user2" ], "owners":[ "user1" ], "runners":[ "user2" ], "writers":[ "user2" ] }} ### Set note permission Description This ```PUT``` method set note authorization information. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/permissions``` Success code 200 Forbidden code 403 Fail code 500 sample JSON input { "readers": [ "user1" ], "owners": [ "user2" ], "runners":[ "user2" ], "writers": [ "user1" ]} sample JSON response { "status": "OK"} ",
"url": " /usage/rest_api/notebook.html",
"group": "usage/rest_api",
"excerpt": "This page contains Apache Zeppelin Notebook REST API information."
}
,
"/usage/rest_api/notebook_repository.html": {
"title": "Apache Zeppelin notebook repository REST API",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Apache Zeppelin Notebook Repository API## OverviewApache Zeppelin provides several REST APIs for interaction and remote activation of zeppelin functionality.All REST APIs are available starting with the following endpoint `http://[zeppelin-server]:[zeppelin-port]/api`. Note that Apache Zeppelin REST APIs receive or return JSON objects, it is recommended for you to install some JSON viewers such as [JSONView](https://chrome.google.com/webstore/detail/jsonview/chklaanhfefbnpoihckbnefhakgolnmc).If you work with Apache Zeppelin and find a need for an additional REST API, please [file an issue or send us an email](http://zeppelin.apache.org/community.html).## Notebook Repository REST API List### List all available notebook repositories Description This ```GET``` method returns all the available notebook repositories. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook-repositories``` Success code 200 Fail code 500 Sample JSON response { "status": "OK", "message": "", "body": [ { "name": "GitNotebookRepo", "className": "org.apache.zeppelin.notebook.repo.GitNotebookRepo", "settings": [ { "type": "INPUT", "value": [], "selected": "ZEPPELIN_HOME/zeppelin/notebook/", "name": "Notebook Path" } ] } ]} ### Reload a notebook repository Description This ```GET``` method triggers reloading and broadcasting of the note list. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook-repositories/reload``` Success code 200 Fail code 500 Sample JSON response { "status": "OK", "message": ""} ### Update a specific notebook repository Description This ```PUT``` method updates a specific notebook repository. URL ```http://[zeppelin-server]:[zeppelin-port]/api/notebook-repositories``` Success code 200 Fail code 404 when the specified notebook repository doesn't exist 406 for invalid payload 500 for any other errors Sample JSON input { "name":"org.apache.zeppelin.notebook.repo.GitNotebookRepo", "settings":{ "Notebook Path":"/tmp/notebook/" }} Sample JSON response { "status": "OK", "message": "", "body": { "name": "GitNotebookRepo", "className": "org.apache.zeppelin.notebook.repo.GitNotebookRepo", "settings": [ { "type": "INPUT", "value": [], "selected": "/tmp/notebook/", "name": "Notebook Path" } ] }} ",
"url": " /usage/rest_api/notebook_repository.html",
"group": "usage/rest_api",
"excerpt": "This page contains Apache Zeppelin notebook repository REST API information."
}
,
"/usage/rest_api/configuration.html": {
"title": "Apache Zeppelin Configuration REST API",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Apache Zeppelin Configuration REST API## OverviewApache Zeppelin provides several REST APIs for interaction and remote activation of zeppelin functionality.All REST APIs are available starting with the following endpoint `http://[zeppelin-server]:[zeppelin-port]/api`. Note that Apache Zeppelin REST APIs receive or return JSON objects, it is recommended for you to install some JSON viewers such as [JSONView](https://chrome.google.com/webstore/detail/jsonview/chklaanhfefbnpoihckbnefhakgolnmc).If you work with Apache Zeppelin and find a need for an additional REST API, please [file an issue or send us an email](http://zeppelin.apache.org/community.html).## Configuration REST API list### List all key/value pair of configurations Description This ```GET``` method return all key/value pair of configurations on the server. Note: For security reason, some pairs would not be shown. URL ```http://[zeppelin-server]:[zeppelin-port]/api/configurations/all``` Success code 200 Fail code 500 sample JSON response { "status": "OK", "message": "", "body": { "zeppelin.war.tempdir": "webapps", "zeppelin.notebook.homescreen.hide": "false", "zeppelin.interpreter.remoterunner": "bin/interpreter.sh", "zeppelin.notebook.s3.user": "user", "zeppelin.server.port": "8089", "zeppelin.dep.localrepo": "local-repo", "zeppelin.ssl.truststore.type": "JKS", "zeppelin.ssl.keystore.path": "keystore", "zeppelin.notebook.s3.bucket": "zeppelin", "zeppelin.server.addr": "0.0.0.0", "zeppelin.ssl.client.auth": "false", "zeppelin.server.context.path": "/", "zeppelin.ssl.keystore.type": "JKS", "zeppelin.ssl.truststore.path": "truststore", "zeppelin.interpreters": "org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkRInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.flink.FlinkInterpreter,org.apache.zeppelin.lens.LensInterpreter,org.apache.zeppelin.ignite.IgniteInterpreter,org.apache.zeppelin.ignite.IgniteSqlInterpreter,org.apache.zeppelin.cassandra.CassandraInterpreter,org.apache.zeppelin.geode.GeodeOqlInterpreter,org.apache.zeppelin.kylin.KylinInterpreter,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter,org.apache.zeppelin.scalding.ScaldingInterpreter", "zeppelin.ssl": "false", "zeppelin.notebook.autoInterpreterBinding": "true", "zeppelin.notebook.homescreen": "", "zeppelin.notebook.storage": "org.apache.zeppelin.notebook.repo.VFSNotebookRepo", "zeppelin.interpreter.connect.timeout": "30000", "zeppelin.anonymous.allowed": "true", "zeppelin.server.allowed.origins":"*", "zeppelin.encoding": "UTF-8" }} ### List all prefix matched key/value pair of configurations Description This ```GET``` method return all prefix matched key/value pair of configurations on the server. Note: For security reason, some pairs would not be shown. URL ```http://[zeppelin-server]:[zeppelin-port]/api/configurations/prefix/[prefix]``` Success code 200 Fail code 500 sample JSON response { "status": "OK", "message": "", "body": { "zeppelin.ssl.keystore.type": "JKS", "zeppelin.ssl.truststore.path": "truststore", "zeppelin.ssl.truststore.type": "JKS", "zeppelin.ssl.keystore.path": "keystore", "zeppelin.ssl": "false", "zeppelin.ssl.client.auth": "false" }} ",
"url": " /usage/rest_api/configuration.html",
"group": "usage/rest_api",
"excerpt": "This page contains Apache Zeppelin Configuration REST API information."
}
,
"/usage/rest_api/helium.html": {
"title": "Apache Zeppelin Helium REST API",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Apache Zeppelin Helium REST API## OverviewApache Zeppelin provides several REST APIs for interaction and remote activation of zeppelin functionality.All REST APIs are available starting with the following endpoint `http://[zeppelin-server]:[zeppelin-port]/api`. Note that Apache Zeppelin REST APIs receive or return JSON objects, it is recommended for you to install some JSON viewers such as [JSONView](https://chrome.google.com/webstore/detail/jsonview/chklaanhfefbnpoihckbnefhakgolnmc).If you work with Apache Zeppelin and find a need for an additional REST API, please [file an issue or send us an email](http://zeppelin.apache.org/community.html).## Helium REST API List### Get all available helium packages Description This ```GET``` method returns all the available helium packages in configured registries. URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/package``` Success code 200 Fail code 500 Sample JSON response { "status": "OK", "message": "", "body": { "zeppelin.clock": [ { "registry": "local", "pkg": { "type": "APPLICATION", "name": "zeppelin.clock", "description": "Clock (example)", "artifact": "zeppelin-examples/zeppelin-example-clock/target/zeppelin-example-clock-0.7.0-SNAPSHOT.jar", "className": "org.apache.zeppelin.example.app.clock.Clock", "resources": [ [ ":java.util.Date" ] ], "icon": "icon" }, "enabled": false } ] }} ### Get all enabled helium packages Description This ```GET``` method returns all enabled helium packages in configured registries. URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/enabledPackage``` Success code 200 Fail code 500 Sample JSON response { "status": "OK", "message": "", "body": { "zeppelin.clock": [ { "registry": "local", "pkg": { "type": "APPLICATION", "name": "zeppelin.clock", "description": "Clock (example)", "artifact": "zeppelin-examples/zeppelin-example-clock/target/zeppelin-example-clock-0.7.0-SNAPSHOT.jar", "className": "org.apache.zeppelin.example.app.clock.Clock", "resources": [ [ ":java.util.Date" ] ], "icon": "icon" }, "enabled": false } ] }} ### Get single helium package Description This ```GET``` method returns specified helium package information URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/package/[Package Name]``` Success code 200 Fail code 500 Sample JSON response { "status": "OK", "message": "", "body": { "zeppelin.clock": [ { "registry": "local", "pkg": { "type": "APPLICATION", "name": "zeppelin.clock", "description": "Clock (example)", "artifact": "zeppelin-examples/zeppelin-example-clock/target/zeppelin-example-clock-0.7.0-SNAPSHOT.jar", "className": "org.apache.zeppelin.example.app.clock.Clock", "resources": [ [ ":java.util.Date" ] ], "icon": "icon" }, "enabled": false } ] }} ### Suggest Helium package on a paragraph Description This ```GET``` method returns suggested helium package for the paragraph. URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/suggest/[Note ID]/[Paragraph ID]``` Success code 200 Fail code 404 on note or paragraph not exists 500 Sample JSON response { "status": "OK", "message": "", "body": { "available": [ { "registry": "local", "pkg": { "type": "APPLICATION", "name": "zeppelin.clock", "description": "Clock (example)", "artifact": "zeppelin-examples/zeppelin-example-clock/target/zeppelin-example-clock-0.7.0-SNAPSHOT.jar", "className": "org.apache.zeppelin.example.app.clock.Clock", "resources": [ [ ":java.util.Date" ] ], "icon": "icon" }, "enabled": true } ] }} ### Load Helium package on a paragraph Description This ```POST``` method loads helium package to target paragraph. URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/load/[Note ID]/[Paragraph ID]``` Success code 200 Fail code 404 on note or paragraph not exists 500 Sample JSON response { "status": "OK", "message": "", "body": "app_2C5FYRZ1E-20170108-040449_2068241472zeppelin_clock"} ### Load bundled visualization script Description This ```GET``` method returns bundled helium visualization javascript. When refresh=true (optional) is provided, Zeppelin rebuilds bundle. Otherwise, it's provided from cache URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/bundle/load/[Package Name][?refresh=true]``` Success code 200 reponse body is executable javascript Fail code 200 reponse body is error message string starts with ERROR: ### Enable package Description This ```POST``` method enables a helium package. Needs artifact name in input payload URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/enable/[Package Name]``` Success code 200 Fail code 500 Sample input zeppelin-examples/zeppelin-example-clock/target/zeppelin-example-clock-0.7.0-SNAPSHOT.jar Sample JSON response {"status":"OK"} ### Disable package Description This ```POST``` method disables a helium package. URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/disable/[Package Name]``` Success code 200 Fail code 500 Sample JSON response {"status":"OK"} ### Get visualization display order Description This ```GET``` method returns display order of enabled visualization packages. URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/order/visualization``` Success code 200 Fail code 500 Sample JSON response {"status":"OK","body":["zeppelin_horizontalbar","zeppelin-bubblechart"]} ### Set visualization display order Description This ```POST``` method sets visualization packages display order. URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/order/visualization``` Success code 200 Fail code 500 Sample JSON input ["zeppelin-bubblechart", "zeppelin_horizontalbar"] Sample JSON response {"status":"OK"} ### Get configuration for all Helium packages Description This ```GET``` method returns configuration for all Helium packages URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/config``` Success code 200 Fail code 500 ### Get configuration for specific package Description This ```GET``` method returns configuration for the specified package name and artifact URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/config/[Package Name]/[Artifact]``` Success code 200 Fail code 500 ### Set configuration for specific package Description This ```POST``` method updates configuration for specified package name and artifact URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/config/[Package Name]/[Artifact]``` Success code 200 Fail code 500 ### Get Spell configuration for single package Description This ```GET``` method returns specified package Spell configuration URL ```http://[zeppelin-server]:[zeppelin-port]/api/helium/spell/config/[Package Name]``` Success code 200 Fail code 500 ",
"url": " /usage/rest_api/helium.html",
"group": "usage/rest_api",
"excerpt": "This page contains Apache Zeppelin Helium REST API information."
}
,
"/quickstart/python_with_zeppelin.html": {
"title": "Python with Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Python support in Zeppelin The following guides explain how to use Apache Zeppelin that enables you to write in Python:- supports flexible python environments using [conda](../interpreter/python.html#conda), [docker](../interpreter/python.html#docker) - can query using [PandasSQL](../interpreter/python.html#sql-over-pandas-dataframes)- also, provides [PySpark](../interpreter/spark.html)- with [matplotlib integration](../interpreter/python.html#matplotlib-integration)- can create results including **UI widgets** using [Dynamic Form](../interpreter/python.html#using-zeppelin-dynamic-forms)For the further information about Spark support in Zeppelin, please check - [Python Interpreter](../interpreter/python.html)",
"url": " /quickstart/python_with_zeppelin.html",
"group": "quickstart",
"excerpt": ""
}
,
"/quickstart/explore_ui.html": {
"title": "Explore Apache Zeppelin UI",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Explore Apache Zeppelin UI## Main homeThe first time you connect to Zeppelin ([default installations start on http://localhost:8080](http://localhost:8080/)), you'll land at the main page similar to the below screen capture.On the left of the page are listed all existing notes. Those notes are stored by default in the `$ZEPPELIN_HOME/notebook` folder.You can filter them by name using the input text form. You can also create a new note, refresh the list of existing notes(in case you manually copy them into the `$ZEPPELIN_HOME/notebook` folder) and import a note.When clicking on `Import Note` link, a new dialog open. From there you can import your note from local disk or from a remote locationif you provide the URL.By default, the name of the imported note is the same as the original note but you can override it by providing a new name.## Menus### NotebookThe `Notebook` menu proposes almost the same features as the note management section in the home page. From the drop-down menu you can:1. Open a selected note2. Filter node by name3. Create a new note### SettingsThis menu gives you access to settings and displays information about Zeppelin. User name is set to `anonymous` if you use default shiro configuration. If you want to set up authentification, see [Shiro Authentication](../setup/security/shiro_authentication.html).#### About ZeppelinYou can check Zeppelin version in this menu.#### InterpreterIn this menu you can:1. Configure existing **interpreter instance**2. Add/remove **interpreter instances**#### CredentialThis menu allows you to save credentials for data sources which are passed to interpreters.#### ConfigurationThis menu displays all the Zeppelin configuration that are set in the config file `$ZEPPELIN_HOME/conf/zeppelin-site.xml`## Note LayoutEach Zeppelin note is composed of 1 .. N paragraphs. The note can be viewed as a paragraph container.### ParagraphEach paragraph consists of 2 sections: `code section` where you put your source code and `result section` where you can see the result of the code execution.On the top-right corner of each paragraph there are some commands to:* execute the paragraph code* hide/show `code section`* hide/show `result section`* configure the paragraphTo configure the paragraph, just click on the gear icon:From this dialog, you can (in descending order):* find the **paragraph id** ( **20150924-163507_134879501** )* control paragraph width. Since Zeppelin is using the grid system of **Twitter Bootstrap**, each paragraph width can be changed from 1 to 12* move the paragraph 1 level up* move the paragraph 1 level down* create a new paragraph* change paragraph title* show/hide line number in the `code section`* disable the run button for this paragraph* export the current paragraph as an **iframe** and open the **iframe** in a new window* clear the `result section`* delete the current paragraph### Note toolbarAt the top of the note, you can find a toolbar which exposes command buttons as well as configuration, security and display options.On the far right is displayed the note name, just click on it to reveal the input form and update it.In the middle of the toolbar you can find the command buttons:* execute all the paragraphs **sequentially**, in their display order* hide/show `code section` of all paragraphs* hide/show `result section` of all paragraphs* clear the `result section` of all paragraphs* clone the current note* export the current note to a JSON file. _Please note that the `code section` and `result section` of all paragraphs will be exported. If you have heavy data in the `result section` of some paragraphs, it is recommended to clean them before exporting* commit the current node content* delete the note* schedule the execution of **all paragraph** using a CRON syntaxOn the right of the note tool bar you can find configuration icons:* display all the keyboard shorcuts* configure the interpreters binding to the current note* configure the note permissions* switch the node display mode between `default`, `simple` and `report`",
"url": " /quickstart/explore_ui.html",
"group": "quickstart",
"excerpt": "If you are new to Apache Zeppelin, this document will guide you about the basic components of Zeppelin one by one."
}
,
"/quickstart/spark_with_zeppelin.html": {
"title": "Spark with Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Spark support in Zeppelin For a brief overview of Apache Spark fundamentals with Apache Zeppelin, see the following guide:- **built-in** Apache Spark integration.- with [SparkSQL](http://spark.apache.org/sql/), [PySpark](https://spark.apache.org/docs/latest/api/python/pyspark.html), [SparkR](https://spark.apache.org/docs/latest/sparkr.html)- inject [SparkContext](https://spark.apache.org/docs/latest/api/java/org/apache/spark/SparkContext.html) and [SQLContext](https://spark.apache.org/docs/latest/sql-programming-guide.html) automatically- dependencies loading (jars) at runtime using [dependency loader](../interpreter/spark.html#dependencyloading) - canceling job and displaying its progress - supporting [Spark Cluster Mode](../setup/deployment/spark_cluster_mode.html#apache-zeppelin-on-spark-cluster-mode) for external spark clusters- supports [different context per user / note](../usage/interpreter/interpreter_binding_mode.html) - sharing variables among PySpark, SparkR and Spark through [ZeppelinContext](../interpreter/spark.html#zeppelincontext)- [Livy Interpreter](../interpreter/livy.html)For the further information about Spark support in Zeppelin, please check - [Spark Interpreter](../interpreter/spark.html)",
"url": " /quickstart/spark_with_zeppelin.html",
"group": "quickstart",
"excerpt": ""
}
,
"/quickstart/install.html": {
"title": "Install",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Install Welcome to Apache Zeppelin! On this page are instructions to help you get started.## Requirements Apache Zeppelin officially supports and is tested on the following environments: Name Value Oracle JDK 1.7 (set JAVA_HOME) OS Mac OSX Ubuntu 14.X CentOS 6.X Windows 7 Pro SP1 ### Downloading Binary PackageTwo binary packages are available on the [download page](http://zeppelin.apache.org/download.html). Only difference between these two binaries is interpreters are included in the package file.- **all interpreter package**: unpack it in a directory of your choice and you're ready to go.- **net-install interpreter package**: unpack and follow [install additional interpreters](../usage/interpreter/installation.html) to install interpreters. If you're unsure, just run `./bin/install-interpreter.sh --all` and install all interpreters. ### Building Zeppelin from sourcefollow the instructions [How to Build](../setup/basics/how_to_build.html), If you want to build from source instead of using binary package.## Starting Apache Zeppelin#### Starting Apache Zeppelin from the Command LineOn all unix like platforms:```bin/zeppelin-daemon.sh start```If you are on Windows:```binzeppelin.cmd```After Zeppelin has started successfully, go to [http://localhost:8080](http://localhost:8080) with your web browser.#### Stopping Zeppelin```bin/zeppelin-daemon.sh stop```## Start Apache Zeppelin with a service manager> **Note :** The below description was written based on Ubuntu.Apache Zeppelin can be auto-started as a service with an init script, using a service manager like **upstart**.This is an example upstart script saved as `/etc/init/zeppelin.conf`This allows the service to be managed with commands such as```sudo service zeppelin start sudo service zeppelin stop sudo service zeppelin restart```Other service managers could use a similar approach with the `upstart` argument passed to the `zeppelin-daemon.sh` script.```bin/zeppelin-daemon.sh upstart```**zeppelin.conf**```description "zeppelin"start on (local-filesystems and net-device-up IFACE!=lo)stop on shutdown# Respawn the process on unexpected terminationrespawn# respawn the job up to 7 times within a 5 second period.# If the job exceeds these values, it will be stopped and marked as failed.respawn limit 7 5# zeppelin was installed in /usr/share/zeppelin in this examplechdir /usr/share/zeppelinexec bin/zeppelin-daemon.sh upstart```## Next StepsCongratulations, you have successfully installed Apache Zeppelin! Here are few steps you might find useful:#### New to Apache Zeppelin... * For an in-depth overview, head to [Explore Zeppelin UI](../quickstart/explore_ui.html). * And then, try run [Tutorial Notebook](http://localhost:8080/#/notebook/2A94M5J1Z) in your Zeppelin. * And see how to change [configurations](../setup/operation/configuration.html) like port number, etc.#### Spark, Python, SQL, and more * [Spark support in Zeppelin](./spark_with_zeppelin.html), to know more about deep integration with [Apache Spark](http://spark.apache.org/). * [SQL support in Zeppelin](./sql_with_zeppelin.html) for SQL support * [Python support in Zeppelin](./python_with_zeppelin.html), for Matplotlib, Pandas, Conda/Docker integration. * [All Available Interpreters](../#available-interpreters)#### Multi-user support ... * Check [Multi-user support](../setup/basics/multi_user_support.html)",
"url": " /quickstart/install.html",
"group": "quickstart",
"excerpt": "This page will help you get started and will guide you through installing Apache Zeppelin and running it in the command line."
}
,
"/quickstart/sql_with_zeppelin.html": {
"title": "SQL with Zeppelin",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# SQL support in Zeppelin The following guides explain how to use Apache Zeppelin that enables you to write in SQL:- provides [JDBC Interpreter](../interpreter/jdbc.html) which allows you can connect any JDBC data sources seamlessly * [Postgres](../interpreter/jdbc.html#postgres) * [MySQL](../interpreter/jdbc.html#mysql) * [MariaDB](../interpreter/jdbc.html#mariadb) * [AWS Redshift](../interpreter/jdbc.html#redshift) * [Apache Hive](../interpreter/jdbc.html#hive) * [Apache Phoenix](../interpreter/jdbc.html#apache-phoenix) * [Apache Drill](../interpreter/jdbc.html#apache-drill) * [Apache Tajo](../interpreter/jdbc.html#apache-tajo) * and so on - [Spark Interpreter](../interpreter/spark.html) supports [SparkSQL](http://spark.apache.org/sql/)- [Python Interpreter](../interpreter/python.html) supports [pandasSQL](../interpreter/python.html#sql-over-pandas-dataframes) - can create query result including **UI widgets** using [Dynamic Form](../usage/dynamic_form/intro.html) ```sql %sql select age, count(1) value from bank where age < ${maxAge=30} group by age order by age ```For the further information about SQL support in Zeppelin, please check - [JDBC Interpreter](../interpreter/jdbc.html)- [Spark Interpreter](../interpreter/spark.html)- [Python Interpreter](../interpreter/python.html)- [IgniteSQL Interpreter](../interpreter/ignite.html#ignite-sql-interpreter) for [Apache Ignite](https://ignite.apache.org/)- [Kylin Interpreter](../interpreter/kylin.html) for [Apache Kylin](http://kylin.apache.org/)",
"url": " /quickstart/sql_with_zeppelin.html",
"group": "quickstart",
"excerpt": ""
}
,
"/quickstart/tutorial.html": {
"title": "Apache Zeppelin Tutorial",
"content" : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Zeppelin TutorialThis tutorial walks you through some of the fundamental Zeppelin concepts. We will assume you have already installed Zeppelin. If not, please see [here](./install.html) first.Current main backend processing engine of Zeppelin is [Apache Spark](https://spark.apache.org). If you're new to this system, you might want to start by getting an idea of how it processes data to get the most out of Zeppelin.## Tutorial with Local File### Data RefineBefore you start Zeppelin tutorial, you will need to download [bank.zip](http://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank.zip). First, to transform csv format data into RDD of `Bank` objects, run following script. This will also remove header using `filter` function.```scalaval bankText = sc.textFile("yourPath/bank/bank-full.csv")case class Bank(age:Integer, job:String, marital : String, education : String, balance : Integer)// split each line, filter out header (starts with "age"), and map it into Bank case classval bank = bankText.map(s=>s.split(";")).filter(s=>s(0)!=""age"").map( s=>Bank(s(0).toInt, s(1).replaceAll(""", ""), s(2).replaceAll(""", ""), s(3).replaceAll(""", ""), s(5).replaceAll(""", "").toInt ))// convert to DataFrame and create temporal tablebank.toDF().registerTempTable("bank")```### Data RetrievalSuppose we want to see age distribution from `bank`. To do this, run:```sql%sql select age, count(1) from bank where age < 30 group by age order by age```You can make input box for setting age condition by replacing `30` with `${maxAge=30}`.```sql%sql select age, count(1) from bank where age < ${maxAge=30} group by age order by age```Now we want to see age distribution with certain marital status and add combo box to select marital status. Run:```sql%sql select age, count(1) from bank where marital="${marital=single,single|divorced|married}" group by age order by age```## Tutorial with Streaming Data ### Data RefineSince this tutorial is based on Twitter's sample tweet stream, you must configure authentication with a Twitter account. To do this, take a look at [Twitter Credential Setup](https://databricks-training.s3.amazonaws.com/realtime-processing-with-spark-streaming.html#twitter-credential-setup). After you get API keys, you should fill out credential related values(`apiKey`, `apiSecret`, `accessToken`, `accessTokenSecret`) with your API keys on following script.This will create a RDD of `Tweet` objects and register these stream data as a table:```scalaimport org.apache.spark.streaming._import org.apache.spark.streaming.twitter._import org.apache.spark.storage.StorageLevelimport scala.io.Sourceimport scala.collection.mutable.HashMapimport java.io.Fileimport org.apache.log4j.Loggerimport org.apache.log4j.Levelimport sys.process.stringSeqToProcess/** Configures the Oauth Credentials for accessing Twitter */def configureTwitterCredentials(apiKey: String, apiSecret: String, accessToken: String, accessTokenSecret: String) { val configs = new HashMap[String, String] ++= Seq( "apiKey" -> apiKey, "apiSecret" -> apiSecret, "accessToken" -> accessToken, "accessTokenSecret" -> accessTokenSecret) println("Configuring Twitter OAuth") configs.foreach{ case(key, value) => if (value.trim.isEmpty) { throw new Exception("Error setting authentication - value for " + key + " not set") } val fullKey = "twitter4j.oauth." + key.replace("api", "consumer") System.setProperty(fullKey, value.trim) println("tProperty " + fullKey + " set as [" + value.trim + "]") } println()}// Configure Twitter credentialsval apiKey = "xxxxxxxxxxxxxxxxxxxxxxxxx"val apiSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"val accessToken = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"val accessTokenSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"configureTwitterCredentials(apiKey, apiSecret, accessToken, accessTokenSecret)import org.apache.spark.streaming.twitter._val ssc = new StreamingContext(sc, Seconds(2))val tweets = TwitterUtils.createStream(ssc, None)val twt = tweets.window(Seconds(60))case class Tweet(createdAt:Long, text:String)twt.map(status=> Tweet(status.getCreatedAt().getTime()/1000, status.getText())).foreachRDD(rdd=> // Below line works only in spark 1.3.0. // For spark 1.1.x and spark 1.2.x, // use rdd.registerTempTable("tweets") instead. rdd.toDF().registerAsTable("tweets"))twt.printssc.start()```### Data RetrievalFor each following script, every time you click run button you will see different result since it is based on real-time data.Let's begin by extracting maximum 10 tweets which contain the word **girl**.```sql%sql select * from tweets where text like '%girl%' limit 10```This time suppose we want to see how many tweets have been created per sec during last 60 sec. To do this, run:```sql%sql select createdAt, count(1) from tweets group by createdAt order by createdAt```You can make user-defined function and use it in Spark SQL. Let's try it by making function named `sentiment`. This function will return one of the three attitudes( positive, negative, neutral ) towards the parameter.```scaladef sentiment(s:String) : String = { val positive = Array("like", "love", "good", "great", "happy", "cool", "the", "one", "that") val negative = Array("hate", "bad", "stupid", "is") var st = 0; val words = s.split(" ") positive.foreach(p => words.foreach(w => if(p==w) st = st+1 ) ) negative.foreach(p=> words.foreach(w=> if(p==w) st = st-1 ) ) if(st>0) "positivie" else if(st<0) "negative" else "neutral"}// Below line works only in spark 1.3.0.// For spark 1.1.x and spark 1.2.x,// use sqlc.registerFunction("sentiment", sentiment _) instead.sqlc.udf.register("sentiment", sentiment _)```To check how people think about girls using `sentiment` function we've made above, run this:```sql%sql select sentiment(text), count(1) from tweets where text like '%girl%' group by sentiment(text)```",
"url": " /quickstart/tutorial.html",
"group": "quickstart",
"excerpt": "This tutorial page contains a short walk-through tutorial that uses Apache Spark backend. Please note that this tutorial is valid for Spark 1.3 and higher."
}
,
}