{
  
  
  

    "/development/howtocontribute.html": {
      "title": "Contributing to Apache Zeppelin (Code)",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Contributing to Apache Zeppelin ( Code )NOTE : Apache Zeppelin is an Apache2 License Software.Any contributions to Zeppelin (Source code, Documents, Image, Website) means you agree with license all your contributions as Apache2 License.Setting upHere are some tools you will need to build and test Zeppelin.Software Configuration Management ( SCM )Since Zeppelin uses Git for it's SCM system, you need git client installed in your development machine.Integrated Development Environment ( IDE )You are free to use whatever IDE you prefer, or your favorite command line editor.Build ToolsTo build the code, installOracle Java 7Apache MavenGetting the source codeFirst of all, you need Zeppelin source code. The official location of Zeppelin is http://git.apache.org/zeppelin.git.git accessGet the source code on your development machine using git.git clone git://git.apache.org/zeppelin.git zeppelinYou may also want to develop against a specific branch. For example, for branch-0.5.6git clone -b branch-0.5.6 git://git.apache.org/zeppelin.git zeppelinApache Zeppelin follows Fork & Pull as a source control workflow.If you want to not only build Zeppelin but also make any changes, then you need to fork Zeppelin github mirror repository and make a pull request.Buildmvn installTo skip testmvn install -DskipTestsTo build with specific spark / hadoop versionmvn install -Dspark.version=x.x.x -Dhadoop.version=x.x.xFor the further Run Zeppelin server in development modecd zeppelin-serverHADOOP_HOME=YOUR_HADOOP_HOME JAVA_HOME=YOUR_JAVA_HOME mvn exec:java -Dexec.mainClass="org.apache.zeppelin.server.ZeppelinServer" -Dexec.args=""Note: Make sure you first run mvn clean install -DskipTests on your zeppelin root directory, otherwise your server build will fail to find the required dependencies in the local repro.or use daemon scriptbin/zeppelin-daemon startServer will be run on http://localhost:8080.Generating Thrift CodeSome portions of the Zeppelin code are generated by Thrift. For most Zeppelin changes, you don't need to worry about this. But if you modify any of the Thrift IDL files (e.g. zeppelin-interpreter/src/main/thrift/*.thrift), then you also need to regenerate these files and submit their updated version as part of your patch.To regenerate the code, install thrift-0.9.2 and then run the following command to generate thrift code.cd <zeppelin_home>/zeppelin-interpreter/src/main/thrift./genthrift.shWhere to StartYou can find issues for beginner & newbieStay involvedContributors should join the Zeppelin mailing lists.dev@zeppelin.apache.org is for people who want to contribute code to Zeppelin. subscribe, unsubscribe, archivesIf you have any issues, create a ticket in JIRA.",
      "url": " /development/howtocontribute.html",
      "group": "development",
      "excerpt": "How can you contribute to Apache Zeppelin project? This document covers from setting up your develop environment to making a pull request on Github."
    }
    ,
    
  

    "/development/howtocontributewebsite.html": {
      "title": "Contributing to Apache Zeppelin (Website)",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Contributing to Apache Zeppelin ( Website )This page will give you an overview of how to build and contribute to the documentation of Apache Zeppelin.The online documentation at zeppelin.apache.org is also generated from the files found here.NOTE : Apache Zeppelin is an Apache2 License Software.Any contributions to Zeppelin (Source code, Documents, Image, Website) means you agree with license all your contributions as Apache2 License.Getting the source codeFirst of all, you need Zeppelin source code. The official location of Zeppelin is http://git.apache.org/zeppelin.git.Documentation website is hosted in 'master' branch under /docs/ dir.git accessFirst of all, you need the website source code. The official location of mirror for Zeppelin is http://git.apache.org/zeppelin.git.Get the source code on your development machine using git.git clone git://git.apache.org/zeppelin.gitcd docsApache Zeppelin follows Fork & Pull as a source control workflow.If you want to not only build Zeppelin but also make any changes, then you need to fork Zeppelin github mirror repository and make a pull request.BuildYou'll need to install some prerequisites to build the code. Please check Build documentation section in docs/README.md.Run website in development modeWhile you're modifying website, you might want to see preview of it. Please check Run website section in docs/README.md.Then you'll be able to access it on http://localhost:4000 with your web browser.Making a Pull RequestWhen you are ready, just make a pull-request.Alternative wayYou can directly edit .md files in /docs/ directory at the web interface of github and make pull-request immediately.Stay involvedContributors should join the Zeppelin mailing lists.dev@zeppelin.apache.org is for people who want to contribute code to Zeppelin. subscribe, unsubscribe, archivesIf you have any issues, create a ticket in JIRA.",
      "url": " /development/howtocontributewebsite.html",
      "group": "development",
      "excerpt": "How can you contribute to Apache Zeppelin project website? This document covers from building Zeppelin documentation site to making a pull request on Github."
    }
    ,
    
  

    "/development/writingzeppelinapplication.html": {
      "title": "Writing a new Application(Experimental)",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Writing a new Application (Experimental)What is Apache Zeppelin ApplicationApache Zeppelin Application is a package that runs on Interpreter process and displays it's output inside of the notebook. While application runs on Interpreter process, it's able to access resources provided by Interpreter through ResourcePool. Output is always rendered by AngularDisplaySystem. Therefore application provides all the possiblities of making interactive graphical application that uses data and processing power of any Interpreter.Make your own ApplicationWriting Application means extending org.apache.zeppelin.helium.Application. You can use your favorite IDE and language while Java class files are packaged into jar. Application class looks like/** * Constructor. Invoked when application is loaded */public Application(ApplicationContext context);/** * Invoked when there're (possible) updates in required resource set. * i.e. invoked after application load and after paragraph finishes. */public abstract void run(ResourceSet args);/** * Invoked before application unload. * Application is automatically unloaded with paragraph/notebook removal */public abstract void unload();You can check example applications under ./zeppelin-examples directory.Development modeIn the development mode, you can run your Application in your IDE as a normal java application and see the result inside of Zeppelin notebook.org.apache.zeppelin.helium.ZeppelinApplicationDevServer can run Zeppelin Application in development mode.// entry point for development modepublic static void main(String[] args) throws Exception {  // add resources for development mode  LocalResourcePool pool = new LocalResourcePool("dev");  pool.put("date", new Date());  // run application in devlopment mode with given resource  // in this case, Clock.class.getName() will be the application class name    org.apache.zeppelin.helium.ZeppelinApplicationDevServer devServer = new org.apache.zeppelin.helium.ZeppelinApplicationDevServer(    Clock.class.getName(), pool.getAll());  // start development mode  devServer.start();  devServer.join();}In the Zeppelin notebook, run %dev run will connect to application running in development mode.Package filePackage file is a json file that provides information about the application.Json file contains the following information{  name : "[organization].[name]",  description : "Description",  artifact : "groupId:artifactId:version",  className : "your.package.name.YourApplicationClass",  resources : [    ["resource.name", ":resource.class.name"],    ["alternative.resource.name", ":alternative.class.name"]  ],  icon : "<i class="icon"></i>"}nameName is a string in [group].[name] format.[group] and [name] allow only [A-Za-z0-9_].Group is normally the name of an organization who creates this application.descriptionA short description about the applicationartifactLocation of the jar artifact."groupId:artifactId:version" will load artifact from maven repository.If jar exists in the local filesystem, absolute/relative can be used.e.g.When artifact exists in Maven repositoryartifact: "org.apache.zeppelin:zeppelin-examples:0.6.0"When artifact exists in the local filesystemartifact: "zeppelin-example/target/zeppelin-example-0.6.0.jar"classNameEntry point. Class that extends org.apache.zeppelin.helium.ApplicationresourcesTwo dimensional array that defines required resources by name or by className. Helium Application launcher will compare resources in the ResourcePool with the information in this field and suggest application only when all required resources are available in the ResourcePool.Resouce name is a string which will be compared with the name of objects in the ResourcePool. className is a string with ":" prepended, which will be compared with className of the objects in the ResourcePool.Application may require two or more resources. Required resources can be listed inside of the json array. For example, if the application requires object "name1", "name2" and "className1" type of object to run, resources field can beresources: [  [ "name1", "name2", ":className1", ...]]If Application can handle alternative combination of required resources, alternative set can be listed as below.resources: [  [ "name", ":className"],  [ "altName", ":altClassName1"],  ...]Easier way to understand this scheme isresources: [   [ 'resource' AND 'resource' AND ... ] OR   [ 'resource' AND 'resource' AND ... ] OR   ...]iconIcon to be used on the application button. String in this field will be rendered as a HTML tag.e.g.icon: "<i class='fa fa-clock-o'></i>"",
      "url": " /development/writingzeppelinapplication.html",
      "group": "development",
      "excerpt": "Apache Zeppelin Application is a package that runs on Interpreter process and displays it's output inside of the notebook. Make your own Application in Apache Zeppelin is quite easy."
    }
    ,
    
  

    "/development/writingzeppelininterpreter.html": {
      "title": "Writing a New Interpreter",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Writing a New InterpreterWhat is Apache Zeppelin InterpreterApache Zeppelin Interpreter is a language backend. For example to use scala code in Zeppelin, you need a scala interpreter.Every Interpreters belongs to an InterpreterGroup.Interpreters in the same InterpreterGroup can reference each other. For example, SparkSqlInterpreter can reference SparkInterpreter to get SparkContext from it while they're in the same group.InterpreterSetting is configuration of a given InterpreterGroup and a unit of start/stop interpreter.All Interpreters in the same InterpreterSetting are launched in a single, separate JVM process. The Interpreter communicates with Zeppelin engine via Thrift.In 'Separate Interpreter(scoped / isolated) for each note' mode which you can see at the Interpreter Setting menu when you create a new interpreter, new interpreter instance will be created per note. But it still runs on the same JVM while they're in the same InterpreterSettings.Make your own InterpreterCreating a new interpreter is quite simple. Just extend org.apache.zeppelin.interpreter abstract class and implement some methods.You can include org.apache.zeppelin:zeppelin-interpreter:[VERSION] artifact in your build system. And you should put your jars under your interpreter directory with a specific directory name. Zeppelin server reads interpreter directories recursively and initializes interpreters including your own interpreter.There are three locations where you can store your interpreter group, name and other information. Zeppelin server tries to find the location below. Next, Zeppelin tries to find interpreter-setting.json in your interpreter jar.{ZEPPELIN_INTERPRETER_DIR}/{YOUR_OWN_INTERPRETER_DIR}/interpreter-setting.jsonHere is an example of interpreter-setting.json on your own interpreter.[  {    "group": "your-group",    "name": "your-name",    "className": "your.own.interpreter.class",    "properties": {      "properties1": {        "envName": null,        "propertyName": "property.1.name",        "defaultValue": "propertyDefaultValue",        "description": "Property description"      },      "properties2": {        "envName": PROPERTIES_2,        "propertyName": null,        "defaultValue": "property2DefaultValue",        "description": "Property 2 description"      }, ...    },    "editor": {      "language": "your-syntax-highlight-language",      "editOnDblClick": false    }  },  {    ...  }]Finally, Zeppelin uses static initialization with the following:static {  Interpreter.register("MyInterpreterName", MyClassName.class.getName());}Static initialization is deprecated and will be supported until 0.6.0.The name will appear later in the interpreter name option box during the interpreter configuration process.The name of the interpreter is what you later write to identify a paragraph which should be interpreted using this interpreter.%MyInterpreterNamesome interpreter specific code...Editor setting for InterpreterYou can add editor object to interpreter-setting.json file to specify paragraph editor settings.LanguageIf the interpreter uses a specific programming language (like Scala, Python, SQL), it is generally recommended to add a syntax highlighting supported for that to the note paragraph editor.To check out the list of languages supported, see the mode-*.js files under zeppelin-web/bower_components/ace-builds/src-noconflict or from github.com/ajaxorg/ace-builds.If you want to add a new set of syntax highlighting,  Add the mode-*.js file to zeppelin-web/bower.json (when built, zeppelin-web/src/index.html will be changed automatically).Add language field to editor object. Note that if you don't specify language field, your interpreter will use plain text mode for syntax highlighting. Let's say you want to set your language to java, then add:"editor": {  "language": "java"}Edit on double clickIf your interpreter uses mark-up language such as markdown or HTML, set editOnDblClick to true so that text editor opens on pargraph double click and closes on paragraph run. Otherwise set it to false."editor": {  "editOnDblClick": false}Install your interpreter binaryOnce you have built your interpreter, you can place it under the interpreter directory with all its dependencies.[ZEPPELIN_HOME]/interpreter/[INTERPRETER_NAME]/Configure your interpreterTo configure your interpreter you need to follow these steps:Add your interpreter class name to the zeppelin.interpreters property in conf/zeppelin-site.xml.Property value is comma separated [INTERPRETER_CLASS_NAME].For example,<property><name>zeppelin.interpreters</name><value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter,com.me.MyNewInterpreter</value></property>Add your interpreter to the default configuration which is used when there is no zeppelin-site.xml.Start Zeppelin by running ./bin/zeppelin-daemon.sh start.In the interpreter page, click the +Create button and configure your interpreter properties.Now you are done and ready to use your interpreter.Note : Interpreters released with zeppelin have a default configuration which is used when there is no conf/zeppelin-site.xml.Use your interpreter0.5.0Inside of a note, %[INTERPRETER_NAME] directive will call your interpreter.Note that the first interpreter configuration in zeppelin.interpreters will be the default one.For example,%myintpval a = "My interpreter"println(a)0.6.0 and laterInside of a note, %[INTERPRETER_GROUP].[INTERPRETER_NAME] directive will call your interpreter.You can omit either [INTERPRETER_GROUP] or [INTERPRETER_NAME]. If you omit [INTERPRETER_NAME], then first available interpreter will be selected in the [INTERPRETER_GROUP].Likewise, if you skip [INTERPRETER_GROUP], then [INTERPRETER_NAME] will be chosen from default interpreter group.For example, if you have two interpreter myintp1 and myintp2 in group mygrp, you can call myintp1 like%mygrp.myintp1codes for myintp1and you can call myintp2 like%mygrp.myintp2codes for myintp2If you omit your interpreter name, it'll select first available interpreter in the group ( myintp1 ).%mygrpcodes for myintp1You can only omit your interpreter group when your interpreter group is selected as a default group.%myintp2codes for myintp2ExamplesCheckout some interpreters released with Zeppelin by default.sparkmarkdownshelljdbcContributing a new Interpreter to Zeppelin releasesWe welcome contribution to a new interpreter. Please follow these few steps:First, check out the general contribution guide here.Follow the steps in Make your own Interpreter section and Editor setting for Interpreter above.Add your interpreter as in the Configure your interpreter section above; also add it to the example template zeppelin-site.xml.template.Add tests! They are run by Travis for all changes and it is important that they are self-contained.Include your interpreter as a module in pom.xml.Add documentation on how to use your interpreter under docs/interpreter/. Follow the Markdown style as this example. Make sure you list config settings and provide working examples on using your interpreter in code boxes in Markdown. Link to images as appropriate (images should go to docs/assets/themes/zeppelin/img/docs-img/). And add a link to your documentation in the navigation menu (docs/_includes/themes/zeppelin/_navigation.html).Most importantly, ensure licenses of the transitive closure of all dependencies are list in license file.Commit your changes and open a Pull Request on the project Mirror on GitHub; check to make sure Travis CI build is passing.",
      "url": " /development/writingzeppelininterpreter.html",
      "group": "development",
      "excerpt": "Apache Zeppelin Interpreter is a language backend. Every Interpreters belongs to an InterpreterGroup. Interpreters in the same InterpreterGroup can reference each other."
    }
    ,
    
  

    "/development/writingzeppelinvisualization.html": {
      "title": "Writing a new Visualization(Experimental)",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Writing a new Visualization (Experimental)What is Apache Zeppelin VisualizationApache Zeppelin Visualization is a pluggable package that can be loaded/unloaded on runtime through Helium framework in Zeppelin. A Visualization is a javascript npm package and user can use them just like any other built-in visualization in notebook.How it works1. Load Helium package files from registryZeppelin needs to know what Visualization packages are available. Zeppelin searches Helium package file from local registry (by default helium/ directory) by default.Helium package file provides informations like name, artifact, and so on. It's similar to package.json in npm package.Here's an example helium/zeppelin-example-horizontalbar.json{  "type" : "VISUALIZATION",  "name" : "zeppelin_horizontalbar",  "description" : "Horizontal Bar chart (example)",  "artifact" : "./zeppelin-examples/zeppelin-example-horizontalbar",  "license" : "Apache-2.0",  "icon" : "<i class='fa fa-bar-chart rotate90flipX'></i>"}Check Create helium package file section to learn about it.2. Enable packagesOnce Zeppelin loads Helium package files from local registry, available packages are displayed in Helium menu.Click 'enable' button.3. Create and load visualization bundle on the flyOnce a Visualization package is enabled, HeliumVisualizationFactory creates a js bundle. The js bundle is served by helium/visualization/load rest api endpoint.4. Run visualizationZeppelin shows additional button for loaded Visualizations.User can use just like any other built-in visualizations.Write new Visualization1. Create a npm packageCreate a package.json in your new Visualization directory. Normally, you can add any dependencies in package.json however Zeppelin Visualization package only allows two dependencies: zeppelin-vis and zeppelin-tabledata.Here's an example{  "name": "zeppelin_horizontalbar",  "description" : "Horizontal Bar chart",  "version": "1.0.0",  "main": "horizontalbar",  "author": "",  "license": "Apache-2.0",  "dependencies": {    "zeppelin-tabledata": "*",    "zeppelin-vis": "*"  }}2. Create your own visualizationTo create your own visualization, you need to create a js file and import Visualization class from zeppelin-vis package and extend the class. zeppelin-tabledata package provides some useful transformations, like pivot, you can use in your visualization. (you can create your own transformation, too).Visualization class, there're several methods that you need to override and implement. Here's simple visualization that just prints Hello world.import Visualization from 'zeppelin-vis'import PassthroughTransformation from 'zeppelin-tabledata/passthrough'export default class helloworld extends Visualization {  constructor(targetEl, config) {    super(targetEl, config)    this.passthrough = new PassthroughTransformation(config);  }  render(tableData) {    this.targetEl.html('Hello world!')  }  getTransformation() {    return this.passthrough  }}To learn more about Visualization class, check visualization.js.You can check complete visualization package example here.Zeppelin's built-in visualization uses the same API, so you can check built-in visualizations as additional examples.3. Create Helium package fileHelium Package file is a json file that provides information about the application.Json file contains the following information{  "type" : "VISUALIZATION",  "name" : "zeppelin_horizontalbar",  "description" : "Horizontal Bar chart (example)",  "license" : "Apache-2.0",  "artifact" : "./zeppelin-examples/zeppelin-example-horizontalbar",  "icon" : "<i class='fa fa-bar-chart rotate90flipX'></i>"}typeWhen you're creating a visualization, 'type' should be 'VISUALIZATION'.Check application type if you're interested in the other types of package.nameName of visualization. Should be unique. Allows [A-Za-z90-9_].descriptionA short description about visualization.artifactLocation of the visualization npm package. Support npm package with version or local filesystem path.e.g.When artifact exists in npm repositoryartifact: "my-visualiztion@1.0.0"When artifact exists in local file systemartifact: "/path/to/my/visualization"licenseLicense information.e.g.license: "Apache-2.0"iconIcon to be used in visualization select button. String in this field will be rendered as a HTML tag.e.g.icon: "<i class='fa fa-coffee'></i>"4. Run in dev modePlace your Helium package file in local registry (ZEPPELIN_HOME/helium).Run Zeppelin. And then run zeppelin-web in visualization dev mode.cd zeppelin-webyarn run visdevYou can browse localhost:9000. Everytime refresh your browser, Zeppelin will rebuild your visualization and reload changes.",
      "url": " /development/writingzeppelinvisualization.html",
      "group": "development",
      "excerpt": "Apache Zeppelin Application is a package that runs on Interpreter process and displays it's output inside of the notebook. Make your own Application in Apache Zeppelin is quite easy."
    }
    ,
    
  

    "/displaysystem/back-end-angular.html": {
      "title": "Back-end Angular API in Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Back-end Angular API in Apache ZeppelinOverviewAngular display system treats output as a view template for AngularJS.It compiles templates and displays them inside of Apache Zeppelin. Zeppelin provides a gateway between your interpreter and your compiled AngularJS view templates.Therefore, you can not only update scope variables from your interpreter but also watch them in the interpreter, which is JVM process.Basic UsagePrint AngularJS viewTo use angular display system, you should start with %angular.Since name is not defined, Hello will display Hello.Please Note: Display system is backend independent.Bind / Unbind VariablesThrough ZeppelinContext, you can bind / unbind variables to AngularJS view. Currently, it only works in Spark Interpreter ( scala ).// bind my 'object' as angular scope variable 'name' in current notebook.z.angularBind(String name, Object object)// bind my 'object' as angular scope variable 'name' in all notebooks related to current interpreter.z.angularBindGlobal(String name, Object object)// unbind angular scope variable 'name' in current notebook.z.angularUnbind(String name)// unbind angular scope variable 'name' in all notebooks related to current interpreter.z.angularUnbindGlobal(String name)Using the above example, let's bind world variable to name. Then you can see AngularJs view is immediately updated.Watch / Unwatch VariablesThrough ZeppelinContext, you can watch / unwatch variables in AngularJs view. Currently, it only works in Spark Interpreter ( scala ).// register for angular scope variable 'name' (notebook)z.angularWatch(String name, (before, after) => { ... })// unregister watcher for angular variable 'name' (notebook)z.angularUnwatch(String name)// register for angular scope variable 'name' (global)z.angularWatchGlobal(String name, (before, after) => { ... })// unregister watcher for angular variable 'name' (global)z.angularUnwatchGlobal(String name)Let's make a button. When it is clicked, the value of run will be increased 1 by 1.z.angularBind("run", 0) will initialize run to zero. And then, it will be also applied to run in z.angularWatch().When the button is clicked, you'll see both run and numWatched are incremented by 1.Let's make it Simpler and more IntuitiveIn this section, we will introduce a simpler and more intuitive way of using Angular Display System in Zeppelin.Here are some usages.Import// In notebook scopeimport org.apache.zeppelin.display.angular.notebookscope._import AngularElem._// In paragraph scopeimport org.apache.zeppelin.display.angular.paragraphscope._import AngularElem._Display Element// automatically convert to string and print with %angular display system directive in front.<div></div>.displayEvent Handler// on click<div></div>.onClick(() => {   my callback routine}).display// on change<div></div>.onChange(() => {  my callback routine}).display// arbitrary event<div></div>.onEvent("ng-click", () => {  my callback routine}).displayBind Model// bind model<div></div>.model("myModel").display// bind model with initial value<div></div>.model("myModel", initialValue).displayInteract with Model// read modelAngularModel("myModel")()// update modelAngularModel("myModel", "newValue")Example: Basic UsageUsing the above basic usages, you can apply them like below examples.Display Elements<div style="color:blue">  <h4>Hello Angular Display System</h4></div>.displayOnClick Event<div class="btn btn-success">  Click me</div>.onClick{() =>  // callback for button click}.displayBind Model  <div>{{{{myModel}}}}</div>.model("myModel", "Initial Value").displayInteract With Model// read the valueAngularModel("myModel")()// update the valueAngularModel("myModel", "New value")Example: String ConverterUsing below example, you can convert the lowercase string to uppercase.// clear previously created angular object.AngularElem.disassociateval button = <div class="btn btn-success btn-sm">Convert</div>.onClick{() =>  val inputString = AngularModel("input")().toString  AngularModel("title", inputString.toUpperCase)}<div>  { <h4> {{{{title}}}}</h4>.model("title", "Please type text to convert uppercase") }   Your text { <input type="text"></input>.model("input", "") }  {button}</div>.display",
      "url": " /displaysystem/back-end-angular.html",
      "group": "display",
      "excerpt": "Apache Zeppelin provides a gateway between your interpreter and your compiled AngularJS view templates. You can not only update scope variables from your interpreter but also watch them in the interpreter, which is JVM process."
    }
    ,
    
  

    "/displaysystem/basicdisplaysystem.html": {
      "title": "Basic Display System in Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Basic Display System in Apache ZeppelinTextBy default, Apache Zeppelin prints interpreter response as a plain text using text display system.You can explicitly say you're using text display system.HtmlWith %html directive, Zeppelin treats your output as HTMLMathematical expressionsHTML display system automatically formats mathematical expression using MathJax. You can use( INLINE EXPRESSION ) and $$ EXPRESSION $$ to format. For exampleTableIf you have data that row separated by n (newline) and column separated by t (tab) with first row as header row, for exampleYou can simply use %table display system to leverage Zeppelin's built in visualization.If table contents start with %html, it is interpreted as an HTML.Note : Display system is backend independent.",
      "url": " /displaysystem/basicdisplaysystem.html",
      "group": "display",
      "excerpt": "There are 3 basic display systems in Apache Zeppelin. By default, Zeppelin prints interpreter responce as a plain text using text display system. With %html directive, Zeppelin treats your output as HTML. You can also simply use %table display system..."
    }
    ,
    
  

    "/displaysystem/front-end-angular.html": {
      "title": "Front-end Angular API in Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Front-end Angular API in Apache ZeppelinBasic UsageIn addition to the back-end API to handle Angular objects binding, Apache Zeppelin also exposes a simple AngularJS z object on the front-end side to expose the same capabilities.This z object is accessible in the Angular isolated scope for each paragraph.Bind / Unbind VariablesThrough the z, you can bind / unbind variables to AngularJS view.Bind a value to an angular object and a mandatory target paragraph:%angular<form class="form-inline">  <div class="form-group">    <label for="superheroId">Super Hero: </label>    <input type="text" class="form-control" id="superheroId" placeholder="Superhero name ..." ng-model="superhero"></input>  </div>  <button type="submit" class="btn btn-primary" ng-click="z.angularBind('superhero',superhero,'20160222-232336_1472609686')"> Bind</button></form>Unbind/remove a value from angular object and a mandatory target paragraph:%angular<form class="form-inline">  <button type="submit" class="btn btn-primary" ng-click="z.angularUnbind('superhero','20160222-232336_1472609686')"> UnBind</button></form>The signature for the z.angularBind() / z.angularUnbind() functions are:// Bindz.angularBind(angularObjectName, angularObjectValue, paragraphId);// Unbindz.angularUnbind(angularObjectName, angularObjectValue, paragraphId);All the parameters are mandatory.Run ParagraphYou can also trigger paragraph execution by calling z.runParagraph() function passing the appropriate paragraphId: %angular<form class="form-inline">  <div class="form-group">    <label for="paragraphId">Paragraph Id: </label>    <input type="text" class="form-control" id="paragraphId" placeholder="Paragraph Id ..." ng-model="paragraph"></input>  </div>  <button type="submit" class="btn btn-primary" ng-click="z.runParagraph(paragraph)"> Run Paragraph</button></form>Overriding dynamic form with Angular ObjectThe front-end Angular Interaction API has been designed to offer richer form capabilities and variable binding. With the existing Dynamic Form system you can already create input text, select and checkbox forms but the choice is rather limited and the look & feel cannot be changed.The idea is to create a custom form using plain HTML/AngularJS code and bind actions on this form to push/remove Angular variables to targeted paragraphs using this new API. Consequently if you use the Dynamic Form syntax in a paragraph and there is a bound Angular object having the same name as the ${formName}, the Angular object will have higher priority and the Dynamic Form will not be displayed. Example: Feature matrix comparisonHow does the front-end AngularJS API compares to the back-end API ? Below is a comparison matrix for both APIs:                        Actions            Front-end API            Back-end API                                Initiate binding            z.angularbind(var, initialValue, paragraphId)            z.angularBind(var, initialValue)                            Update value            same to ordinary angularjs scope variable, or z.angularbind(var, newValue, paragraphId)            z.angularBind(var, newValue)                            Watching value            same to ordinary angularjs scope variable            z.angularWatch(var, (oldVal, newVal) => ...)                            Destroy binding            z.angularUnbind(var, paragraphId)            z.angularUnbind(var)                            Executing Paragraph            z.runParagraph(paragraphId)            z.run(paragraphId)                            Executing Paragraph (Specific paragraphs in other notes) (                        z.run(noteid, paragraphId)                            Executing note                        z.runNote(noteId)                     Both APIs are pretty similar, except for value watching where it is done naturally by AngularJS internals on the front-end and by user custom watcher functions in the back-end.There is also a slight difference in term of scope. Front-end API limits the Angular object binding to a paragraph scope whereas back-end API allows you to bind an Angular object at the global or note scope. This restriction has been designed purposely to avoid Angular object leaks and scope pollution.",
      "url": " /displaysystem/front-end-angular.html",
      "group": "display",
      "excerpt": "In addition to the back-end API to handle Angular objects binding, Apache Zeppelin exposes a simple AngularJS z object on the front-end side to expose the same capabilities."
    }
    ,
    
  
  

    "/install/build.html": {
      "title": "Build from Source",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Building from SourceIf you want to build from source, you must first install the following dependencies:      Name    Value        Git    (Any Version)        Maven    3.1.x or higher        JDK    1.7  If you haven't installed Git and Maven yet, check the Build requirements section and follow the step by step instructions from there.1. Clone the Apache Zeppelin repositorygit clone https://github.com/apache/zeppelin.git2. Build sourceYou can build Zeppelin with following maven command:mvn clean package -DskipTests [Options]If you're unsure about the options, use the same commands that creates official binary package.# update all pom.xml to use scala 2.11./dev/change_scala_version.sh 2.11# build zeppelin with all interpreters and include latest version of Apache spark support for local mode.mvn clean package -DskipTests -Pspark-2.0 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -Pr -Pscala-2.113. DoneYou can directly start Zeppelin by running after successful build:./bin/zeppelin-daemon.sh startCheck build-profiles section for further build options.If you are behind proxy, follow instructions in Proxy setting section.If you're interested in contribution, please check Contributing to Apache Zeppelin (Code) and Contributing to Apache Zeppelin (Website).Build profilesSpark InterpreterTo build with a specific Spark version, Hadoop version or specific features, define one or more of the following profiles and options:-Pspark-[version]Set spark major versionAvailable profiles are-Pspark-2.1-Pspark-2.0-Pspark-1.6-Pspark-1.5-Pspark-1.4-Pcassandra-spark-1.5-Pcassandra-spark-1.4-Pcassandra-spark-1.3-Pcassandra-spark-1.2-Pcassandra-spark-1.1minor version can be adjusted by -Dspark.version=x.x.x-Phadoop-[version]set hadoop major versionAvailable profiles are-Phadoop-0.23-Phadoop-1-Phadoop-2.2-Phadoop-2.3-Phadoop-2.4-Phadoop-2.6-Phadoop-2.7minor version can be adjusted by -Dhadoop.version=x.x.x-Pscala-[version] (optional)set scala version (default 2.10)Available profiles are-Pscala-2.10-Pscala-2.11-Pyarn (optional)enable YARN support for local modeYARN for local mode is not supported for Spark v1.5.0 or higher. Set SPARK_HOME instead.-Ppyspark (optional)enable PySpark support for local mode.-Pr (optional)enable R support with SparkR integration.-Psparkr (optional)another R support with SparkR integration as well as local mode support.-Pvendor-repo (optional)enable 3rd party vendor repository (cloudera)-Pmapr[version] (optional)For the MapR Hadoop Distribution, these profiles will handle the Hadoop version. As MapR allows different versions of Spark to be installed, you should specify which version of Spark is installed on the cluster by adding a Spark profile (-Pspark-1.6, -Pspark-2.0, etc.) as needed.The correct Maven artifacts can be found for every version of MapR at http://doc.mapr.comAvailable profiles are-Pmapr3-Pmapr40-Pmapr41-Pmapr50-Pmapr51-Pexamples (optional)Bulid examples under zeppelin-examples directoryBuild command examplesHere are some examples with several options:# build with spark-2.1, scala-2.11./dev/change_scala_version.sh 2.11mvn clean package -Pspark-2.1 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -Pscala-2.11 -DskipTests# build with spark-2.0, scala-2.11./dev/change_scala_version.sh 2.11mvn clean package -Pspark-2.0 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -Pscala-2.11 -DskipTests# build with spark-1.6, scala-2.10mvn clean package -Pspark-1.6 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -DskipTests# spark-cassandra integrationmvn clean package -Pcassandra-spark-1.5 -Dhadoop.version=2.6.0 -Phadoop-2.6 -DskipTests -DskipTests# with CDHmvn clean package -Pspark-1.5 -Dhadoop.version=2.6.0-cdh5.5.0 -Phadoop-2.6 -Pvendor-repo -DskipTests# with MapRmvn clean package -Pspark-1.5 -Pmapr50 -DskipTestsIgnite Interpretermvn clean package -Dignite.version=1.9.0 -DskipTestsScalding Interpretermvn clean package -Pscalding -DskipTestsBuild requirementsInstall requirementsIf you don't have requirements prepared, install it.(The installation method may vary according to your environment, example is for Ubuntu.)sudo apt-get updatesudo apt-get install gitsudo apt-get install openjdk-7-jdksudo apt-get install npmsudo apt-get install libfontconfigInstall mavenwget http://www.eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gzsudo tar -zxf apache-maven-3.3.9-bin.tar.gz -C /usr/local/sudo ln -s /usr/local/apache-maven-3.3.9/bin/mvn /usr/local/bin/mvnNotes: - Ensure node is installed by running node --version - Ensure maven is running version 3.1.x or higher with mvn -version - Configure maven to use more memory than usual by export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=1024m"Proxy setting (optional)If you're behind the proxy, you'll need to configure maven and npm to pass through it.First of all, configure maven in your ~/.m2/settings.xml.<settings>  <proxies>    <proxy>      <id>proxy-http</id>      <active>true</active>      <protocol>http</protocol>      <host>localhost</host>      <port>3128</port>      <!-- <username>usr</username>      <password>pwd</password> -->      <nonProxyHosts>localhost|127.0.0.1</nonProxyHosts>    </proxy>    <proxy>      <id>proxy-https</id>      <active>true</active>      <protocol>https</protocol>      <host>localhost</host>      <port>3128</port>      <!-- <username>usr</username>      <password>pwd</password> -->      <nonProxyHosts>localhost|127.0.0.1</nonProxyHosts>    </proxy>  </proxies></settings>Then, next commands will configure npm.npm config set proxy http://localhost:3128npm config set https-proxy http://localhost:3128npm config set registry "http://registry.npmjs.org/"npm config set strict-ssl falseConfigure git as wellgit config --global http.proxy http://localhost:3128git config --global https.proxy http://localhost:3128git config --global url."http://".insteadOf git://To clean up, set active false in Maven settings.xml and run these commands.npm config rm proxynpm config rm https-proxygit config --global --unset http.proxygit config --global --unset https.proxygit config --global --unset url."http://".insteadOfNotes: - If you are behind NTLM proxy you can use Cntlm Authentication Proxy. - Replace localhost:3128 with the standard pattern http://user:pwd@host:port.PackageTo package the final distribution including the compressed archive, run:mvn clean package -Pbuild-distrTo build a distribution with specific profiles, run:mvn clean package -Pbuild-distr -Pspark-1.5 -Phadoop-2.4 -Pyarn -PpysparkThe profiles -Pspark-1.5 -Phadoop-2.4 -Pyarn -Ppyspark can be adjusted if you wish to build to a specific spark versions, or omit support such as yarn.  The archive is generated under zeppelin-distribution/target directoryRun end-to-end testsZeppelin comes with a set of end-to-end acceptance tests driving headless selenium browser# assumes zeppelin-server running on localhost:8080 (use -Durl=.. to override)mvn verify# or take care of starting/stoping zeppelin-server from packaged zeppelin-distribuion/targetmvn verify -P using-packaged-distr",
      "url": " /install/build.html",
      "group": "install",
      "excerpt": "How to build Zeppelin from source"
    }
    ,
    
  

    "/install/cdh.html": {
      "title": "Apache Zeppelin on CDH",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Zeppelin on CDH1. Import Cloudera QuickStart Docker imageCloudera has officially provided CDH Docker Hub in their own container. Please check this guide page for more information.You can import the Docker image by pulling it from Cloudera Docker Hub.docker pull cloudera/quickstart:latest2. Run dockerdocker run -it  -p 80:80  -p 4040:4040  -p 8020:8020  -p 8022:8022  -p 8030:8030  -p 8032:8032  -p 8033:8033  -p 8040:8040  -p 8042:8042  -p 8088:8088  -p 8480:8480  -p 8485:8485  -p 8888:8888  -p 9083:9083  -p 10020:10020  -p 10033:10033  -p 18088:18088  -p 19888:19888  -p 25000:25000  -p 25010:25010  -p 25020:25020  -p 50010:50010  -p 50020:50020  -p 50070:50070  -p 50075:50075  -h quickstart.cloudera --privileged=true  agitated_payne_backup /usr/bin/docker-quickstart;3. Verify running CDHTo verify the application is running well, check the web UI for HDFS on http://<hostname>:50070/ and YARN on http://<hostname>:8088/cluster.4. Configure Spark interpreter in ZeppelinSet following configurations to conf/zeppelin-env.sh.export MASTER=yarn-clientexport HADOOP_CONF_DIR=[your_hadoop_conf_path]export SPARK_HOME=[your_spark_home_path]HADOOP_CONF_DIR(Hadoop configuration path) is defined in /scripts/docker/spark-cluster-managers/cdh/hdfs_conf.Don't forget to set Spark master as yarn-client in Zeppelin Interpreters setting page like below.5. Run Zeppelin with Spark interpreterAfter running a single paragraph with Spark interpreter in Zeppelin,browse http://<hostname>:8088/cluster/apps to check Zeppelin application is running well or not.",
      "url": " /install/cdh.html",
      "group": "install",
      "excerpt": "This document will guide you how you can build and configure the environment on CDH with Apache Zeppelin using docker scripts."
    }
    ,
    
  

    "/install/configuration.html": {
      "title": "Apache Zeppelin Configuration",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Zeppelin ConfigurationZeppelin PropertiesThere are two locations you can configure Apache Zeppelin.Environment variables can be defined conf/zeppelin-env.sh(confzeppelin-env.cmd for Windows).Java properties can ba defined in conf/zeppelin-site.xml.If both are defined, then the environment variables will take priority.      zeppelin-env.sh    zeppelin-site.xml    Default value    Description        ZEPPELIN_PORT    zeppelin.server.port    8080    Zeppelin server port        Note: Please make sure you're not using the same port with      Zeppelin web application development port (default: 9000).        ZEPPELIN_SSL_PORT    zeppelin.server.ssl.port    8443    Zeppelin Server ssl port (used when ssl environment/property is set to true)        ZEPPELIN_MEM    N/A    -Xmx1024m -XX:MaxPermSize=512m    JVM mem options        ZEPPELIN_INTP_MEM    N/A    ZEPPELIN_MEM    JVM mem options for interpreter process        ZEPPELIN_JAVA_OPTS    N/A        JVM options        ZEPPELIN_ALLOWED_ORIGINS    zeppelin.server.allowed.origins    *    Enables a way to specify a ',' separated list of allowed origins for REST and websockets.  e.g. http://localhost:8080           N/A    zeppelin.anonymous.allowed    true    The anonymous user is allowed by default.        ZEPPELIN_SERVER_CONTEXT_PATH    zeppelin.server.context.path    /    Context path of the web application        ZEPPELIN_SSL    zeppelin.ssl    false            ZEPPELIN_SSL_CLIENT_AUTH    zeppelin.ssl.client.auth    false            ZEPPELIN_SSL_KEYSTORE_PATH    zeppelin.ssl.keystore.path    keystore            ZEPPELIN_SSL_KEYSTORE_TYPE    zeppelin.ssl.keystore.type    JKS            ZEPPELIN_SSL_KEYSTORE_PASSWORD    zeppelin.ssl.keystore.password                ZEPPELIN_SSL_KEY_MANAGER_PASSWORD    zeppelin.ssl.key.manager.password                ZEPPELIN_SSL_TRUSTSTORE_PATH    zeppelin.ssl.truststore.path                ZEPPELIN_SSL_TRUSTSTORE_TYPE    zeppelin.ssl.truststore.type                ZEPPELIN_SSL_TRUSTSTORE_PASSWORD    zeppelin.ssl.truststore.password                ZEPPELIN_NOTEBOOK_HOMESCREEN    zeppelin.notebook.homescreen        Display note IDs on the Apache Zeppelin homescreen e.g. 2A94M5J1Z        ZEPPELIN_NOTEBOOK_HOMESCREEN_HIDE    zeppelin.notebook.homescreen.hide    false    Hide the note ID set by ZEPPELIN_NOTEBOOK_HOMESCREEN on the Apache Zeppelin homescreen. For the further information, please read Customize your Zeppelin homepage.        ZEPPELIN_WAR_TEMPDIR    zeppelin.war.tempdir    webapps    Location of the jetty temporary directory        ZEPPELIN_NOTEBOOK_DIR    zeppelin.notebook.dir    notebook    The root directory where notebook directories are saved        ZEPPELIN_NOTEBOOK_S3_BUCKET    zeppelin.notebook.s3.bucket    zeppelin    S3 Bucket where notebook files will be saved        ZEPPELIN_NOTEBOOK_S3_USER    zeppelin.notebook.s3.user    user    User name of an S3 buckete.g. bucket/user/notebook/2A94M5J1Z/note.json        ZEPPELIN_NOTEBOOK_S3_ENDPOINT    zeppelin.notebook.s3.endpoint    s3.amazonaws.com    Endpoint for the bucket        ZEPPELIN_NOTEBOOK_S3_KMS_KEY_ID    zeppelin.notebook.s3.kmsKeyID        AWS KMS Key ID to use for encrypting data in S3 (optional)        ZEPPELIN_NOTEBOOK_S3_EMP    zeppelin.notebook.s3.encryptionMaterialsProvider        Class name of a custom S3 encryption materials provider implementation to use for encrypting data in S3 (optional)        ZEPPELIN_NOTEBOOK_AZURE_CONNECTION_STRING    zeppelin.notebook.azure.connectionString        The Azure storage account connection stringe.g. DefaultEndpointsProtocol=https;AccountName=<accountName>;AccountKey=<accountKey>        ZEPPELIN_NOTEBOOK_AZURE_SHARE    zeppelin.notebook.azure.share    zeppelin    Azure Share where the notebook files will be saved        ZEPPELIN_NOTEBOOK_AZURE_USER    zeppelin.notebook.azure.user    user    Optional user name of an Azure file sharee.g. share/user/notebook/2A94M5J1Z/note.json        ZEPPELIN_NOTEBOOK_STORAGE    zeppelin.notebook.storage    org.apache.zeppelin.notebook.repo.GitNotebookRepo    Comma separated list of notebook storage locations        ZEPPELIN_NOTEBOOK_ONE_WAY_SYNC    zeppelin.notebook.one.way.sync    false    If there are multiple notebook storage locations, should we treat the first one as the only source of truth?        ZEPPELIN_NOTEBOOK_PUBLIC    zeppelin.notebook.public    true    Make notebook public (set only owners) by default when created/imported. If set to false will add user to readers and writers as well, making it private and invisible to other users unless permissions are granted.        ZEPPELIN_INTERPRETERS    zeppelin.interpreters      org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,    ...              Comma separated interpreter configurations [Class]       NOTE: This property is deprecated since Zeppelin-0.6.0 and will not be supported from Zeppelin-0.7.0.            ZEPPELIN_INTERPRETER_DIR    zeppelin.interpreter.dir    interpreter    Interpreter directory        ZEPPELIN_INTERPRETER_DEP_MVNREPO    zeppelin.interpreter.dep.mvnRepo    http://repo1.maven.org/maven2/    Remote principal repository for interpreter's additional dependency loading        ZEPPELIN_DEP_LOCALREPO    zeppelin.dep.localrepo    local-repo    Local repository for dependency loader.ex)visualiztion modules of npm.        ZEPPELIN_HELIUM_NPM_REGISTRY    zeppelin.helium.npm.registry    http://registry.npmjs.org/    Remote Npm registry for Helium dependency loader        ZEPPELIN_INTERPRETER_OUTPUT_LIMIT    zeppelin.interpreter.output.limit    102400    Output message from interpreter exceeding the limit will be truncated        ZEPPELIN_WEBSOCKET_MAX_TEXT_MESSAGE_SIZE    zeppelin.websocket.max.text.message.size    1024000    Size (in characters) of the maximum text message that can be received by websocket.  SSL ConfigurationEnabling SSL requires a few configuration changes. First, you need to create certificates and then update necessary configurations to enable server side SSL and/or client side certificate authentication.Creating and configuring the CertificatesInformation how about to generate certificates and a keystore can be found here.A condensed example can be found in the top answer to this StackOverflow post.The keystore holds the private key and certificate on the server end. The trustore holds the trusted client certificates. Be sure that the path and password for these two stores are correctly configured in the password fields below. They can be obfuscated using the Jetty password tool. After Maven pulls in all the dependency to build Zeppelin, one of the Jetty jars contain the Password tool. Invoke this command from the Zeppelin home build directory with the appropriate version, user, and password.java -cp ./zeppelin-server/target/lib/jetty-all-server-<version>.jar org.eclipse.jetty.util.security.Password <user> <password>If you are using a self-signed, a certificate signed by an untrusted CA, or if client authentication is enabled, then the client must have a browser create exceptions for both the normal HTTPS port and WebSocket port. This can by done by trying to establish an HTTPS connection to both ports in a browser (e.g. if the ports are 443 and 8443, then visit https://127.0.0.1:443 and https://127.0.0.1:8443). This step can be skipped if the server certificate is signed by a trusted CA and client auth is disabled.Configuring server side SSLThe following properties needs to be updated in the zeppelin-site.xml in order to enable server side SSL.<property>  <name>zeppelin.server.ssl.port</name>  <value>8443</value>  <description>Server ssl port. (used when ssl property is set to true)</description></property><property>  <name>zeppelin.ssl</name>  <value>true</value>  <description>Should SSL be used by the servers?</description></property><property>  <name>zeppelin.ssl.keystore.path</name>  <value>keystore</value>  <description>Path to keystore relative to Zeppelin configuration directory</description></property><property>  <name>zeppelin.ssl.keystore.type</name>  <value>JKS</value>  <description>The format of the given keystore (e.g. JKS or PKCS12)</description></property><property>  <name>zeppelin.ssl.keystore.password</name>  <value>change me</value>  <description>Keystore password. Can be obfuscated by the Jetty Password tool</description></property><property>  <name>zeppelin.ssl.key.manager.password</name>  <value>change me</value>  <description>Key Manager password. Defaults to keystore password. Can be obfuscated.</description></property>Enabling client side certificate authenticationThe following properties needs to be updated in the zeppelin-site.xml in order to enable client side certificate authentication.<property>  <name>zeppelin.server.ssl.port</name>  <value>8443</value>  <description>Server ssl port. (used when ssl property is set to true)</description></property><property>  <name>zeppelin.ssl.client.auth</name>  <value>true</value>  <description>Should client authentication be used for SSL connections?</description></property><property>  <name>zeppelin.ssl.truststore.path</name>  <value>truststore</value>  <description>Path to truststore relative to Zeppelin configuration directory. Defaults to the keystore path</description></property><property>  <name>zeppelin.ssl.truststore.type</name>  <value>JKS</value>  <description>The format of the given truststore (e.g. JKS or PKCS12). Defaults to the same type as the keystore type</description></property><property>  <name>zeppelin.ssl.truststore.password</name>  <value>change me</value>  <description>Truststore password. Can be obfuscated by the Jetty Password tool. Defaults to the keystore password</description></property>Obfuscating Passwords using the Jetty Password ToolSecurity best practices advise to not use plain text passwords and Jetty provides a password tool to help obfuscating the passwords used to access the KeyStore and TrustStore.The Password tool documentation can be found here.After using the tool:java -cp $ZEPPELIN_HOME/zeppelin-server/target/lib/jetty-util-9.2.15.v20160210.jar          org.eclipse.jetty.util.security.Password           password2016-12-15 10:46:47.931:INFO::main: Logging initialized @101mspasswordOBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1vMD5:5f4dcc3b5aa765d61d8327deb882cf99update your configuration with the obfuscated password :<property>  <name>zeppelin.ssl.keystore.password</name>  <value>OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</value>  <description>Keystore password. Can be obfuscated by the Jetty Password tool</description></property>Note: After updating these configurations, Zeppelin server needs to be restarted.",
      "url": " /install/configuration.html",
      "group": "install",
      "excerpt": "This page will guide you to configure Apache Zeppelin using either environment variables or Java properties. Also, you can configure SSL for Zeppelin."
    }
    ,
    
  

    "/install/docker.html": {
      "title": "Apache Zeppelin Releases Docker Images",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Docker Image for Apache Zeppelin ReleasesOverviewThis document contains instructions about making docker containers for Zeppelin. It mainly provides guidance into how to create, publish and run docker images for zeppelin releases.Quick StartInstalling DockerYou need to install docker on your machine.Running docker imagedocker run -p 8080:8080 --rm --name zeppelin apache/zeppelin:<release-version> Zeppelin will run at http://localhost:8080.If you want to specify logs and notebook dir, docker run -p 8080:8080 --rm -v $PWD/logs:/logs -v $PWD/notebook:/notebook -e ZEPPELIN_LOG_DIR='/logs' -e ZEPPELIN_NOTEBOOK_DIR='/notebook' --name zeppelin apache/zeppelin:<release-version> # e.g '0.7.2'Building dockerfile locallycd $ZEPPELIN_HOMEcd scripts/docker/zeppelin/bindocker build -t my-zeppelin:my-tag ./",
      "url": " /install/docker.html",
      "group": "install",
      "excerpt": "This document contains instructions about making docker containers for Zeppelin. It mainly provides guidance into how to create, publish and run docker images for zeppelin releases."
    }
    ,
    
  

    "/install/install.html": {
      "title": "Quick Start",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Quick StartWelcome to Apache Zeppelin! On this page are instructions to help you get started.InstallationApache Zeppelin officially supports and is tested on the following environments:      Name    Value        Oracle JDK    1.7  (set JAVA_HOME)        OS    Mac OSX  Ubuntu 14.X  CentOS 6.X  Windows 7 Pro SP1  Downloading Binary PackageTwo binary packages are available on the Apache Zeppelin Download Page. Only difference between these two binaries is interpreters are included in the package file.Package with all interpreters.Just unpack it in a directory of your choice and you're ready to go.Package with net-install interpreters.Unpack and follow install additional interpreters to install interpreters. If you're unsure, just run ./bin/install-interpreter.sh --all and install all interpreters.Starting Apache ZeppelinStarting Apache Zeppelin from the Command LineOn all unix like platforms:bin/zeppelin-daemon.sh startIf you are on Windows:binzeppelin.cmdAfter Zeppelin has started successfully, go to http://localhost:8080 with your web browser.Stopping Zeppelinbin/zeppelin-daemon.sh stopStart Apache Zeppelin with a service managerNote : The below description was written based on Ubuntu Linux.Apache Zeppelin can be auto-started as a service with an init script, using a service manager like upstart.This is an example upstart script saved as /etc/init/zeppelin.confThis allows the service to be managed with commands such assudo service zeppelin start  sudo service zeppelin stop  sudo service zeppelin restartOther service managers could use a similar approach with the upstart argument passed to the zeppelin-daemon.sh script.bin/zeppelin-daemon.sh upstartzeppelin.confdescription "zeppelin"start on (local-filesystems and net-device-up IFACE!=lo)stop on shutdown# Respawn the process on unexpected terminationrespawn# respawn the job up to 7 times within a 5 second period.# If the job exceeds these values, it will be stopped and marked as failed.respawn limit 7 5# zeppelin was installed in /usr/share/zeppelin in this examplechdir /usr/share/zeppelinexec bin/zeppelin-daemon.sh upstartNext StepsCongratulations, you have successfully installed Apache Zeppelin! Here are few steps you might find useful:New to Apache Zeppelin...For an in-depth overview, head to Explore Apache Zeppelin UI.And then, try run tutorial notebook in your Zeppelin.And see how to change configurations like port number, etc.Zeppelin with Apache Spark ...To know more about deep integration with Apache Spark, check Spark Interpreter.Zeppelin with JDBC data sources ...Check JDBC Interpreter to know more about configure and uses multiple JDBC data sources.Zeppelin with Python ...Check Python interpreter to know more about Matplotlib, Pandas, Conda/Docker environment integration.Multi-user environment ...Turn on authentication.Manage your notebook permission.For more informations, go to More -> Security section.Other useful informations ...Learn how Display System works.Use Service Manager to start Zeppelin.If you're using previous version please see Upgrade Zeppelin version.Building Apache Zeppelin from SourceIf you want to build from source instead of using binary package, follow the instructions here.",
      "url": " /install/install.html",
      "group": "install",
      "excerpt": "This page will help you get started and will guide you through installing Apache Zeppelin and running it in the command line."
    }
    ,
    
  

    "/install/spark_cluster_mode.html": {
      "title": "Apache Zeppelin on Spark cluster mode",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Zeppelin on Spark Cluster ModeOverviewApache Spark has supported three cluster manager types(Standalone, Apache Mesos and Hadoop YARN) so far.This document will guide you how you can build and configure the environment on 3 types of Spark cluster manager with Apache Zeppelin using Docker scripts.So install docker on the machine first.Spark standalone modeSpark standalone is a simple cluster manager included with Spark that makes it easy to set up a cluster.You can simply set up Spark standalone environment with below steps.Note : Since Apache Zeppelin and Spark use same 8080 port for their web UI, you might need to change zeppelin.server.port in conf/zeppelin-site.xml.1. Build Docker fileYou can find docker script files under scripts/docker/spark-cluster-managers.cd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_standalonedocker build -t "spark_standalone" .2. Run dockerdocker run -it -p 8080:8080 -p 7077:7077 -p 8888:8888 -p 8081:8081 -h sparkmaster --name spark_standalone spark_standalone bash;Note that sparkmaster hostname used here to run docker container should be defined in your /etc/hosts.3. Configure Spark interpreter in ZeppelinSet Spark master as spark://<hostname>:7077 in Zeppelin Interpreters setting page.4. Run Zeppelin with Spark interpreterAfter running single paragraph with Spark interpreter in Zeppelin, browse https://<hostname>:8080 and check whether Spark cluster is running well or not.You can also simply verify that Spark is running well in Docker with below command.ps -ef | grep sparkSpark on YARN modeYou can simply set up Spark on YARN docker environment with below steps.Note : Since Apache Zeppelin and Spark use same 8080 port for their web UI, you might need to change zeppelin.server.port in conf/zeppelin-site.xml.1. Build Docker fileYou can find docker script files under scripts/docker/spark-cluster-managers.cd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_yarn_clusterdocker build -t "spark_yarn" .2. Run dockerdocker run -it  -p 5000:5000  -p 9000:9000  -p 9001:9001  -p 8088:8088  -p 8042:8042  -p 8030:8030  -p 8031:8031  -p 8032:8032  -p 8033:8033  -p 8080:8080  -p 7077:7077  -p 8888:8888  -p 8081:8081  -p 50010:50010  -p 50075:50075  -p 50020:50020  -p 50070:50070  --name spark_yarn  -h sparkmaster  spark_yarn bash;Note that sparkmaster hostname used here to run docker container should be defined in your /etc/hosts.3. Verify running Spark on YARN.You can simply verify the processes of Spark and YARN are running well in Docker with below command.ps -efYou can also check each application web UI for HDFS on http://<hostname>:50070/, YARN on http://<hostname>:8088/cluster and Spark on http://<hostname>:8080/.4. Configure Spark interpreter in ZeppelinSet following configurations to conf/zeppelin-env.sh.export MASTER=yarn-clientexport HADOOP_CONF_DIR=[your_hadoop_conf_path]export SPARK_HOME=[your_spark_home_path]HADOOP_CONF_DIR(Hadoop configuration path) is defined in /scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf.Don't forget to set Spark master as yarn-client in Zeppelin Interpreters setting page like below.5. Run Zeppelin with Spark interpreterAfter running a single paragraph with Spark interpreter in Zeppelin, browse http://<hostname>:8088/cluster/apps and check Zeppelin application is running well or not.Spark on Mesos modeYou can simply set up Spark on Mesos docker environment with below steps.1. Build Docker filecd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_mesosdocker build -t "spark_mesos" .2. Run dockerdocker run --net=host -it -p 8080:8080 -p 7077:7077 -p 8888:8888 -p 8081:8081 -p 8082:8082 -p 5050:5050 -p 5051:5051 -p 4040:4040 -h sparkmaster --name spark_mesos spark_mesos bash;Note that sparkmaster hostname used here to run docker container should be defined in your /etc/hosts.3. Verify running Spark on Mesos.You can simply verify the processes of Spark and Mesos are running well in Docker with below command.ps -efYou can also check each application web UI for Mesos on http://<hostname>:5050/cluster and Spark on http://<hostname>:8080/.4. Configure Spark interpreter in Zeppelinexport MASTER=mesos://127.0.1.1:5050export MESOS_NATIVE_JAVA_LIBRARY=[PATH OF libmesos.so]export SPARK_HOME=[PATH OF SPARK HOME]Don't forget to set Spark master as mesos://127.0.1.1:5050 in Zeppelin Interpreters setting page like below.5. Run Zeppelin with Spark interpreterAfter running a single paragraph with Spark interpreter in Zeppelin, browse http://<hostname>:5050/#/frameworks and check Zeppelin application is running well or not.Troubleshooting for Spark on MesosIf you have problem with hostname, use --add-host option when executing dockerrun## use `--add-host=moby:127.0.0.1` option to resolve## since docker container couldn't resolve `moby`: java.net.UnknownHostException: moby: moby: Name or service not known        at java.net.InetAddress.getLocalHost(InetAddress.java:1496)        at org.apache.spark.util.Utils$.findLocalInetAddress(Utils.scala:789)        at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress$lzycompute(Utils.scala:782)        at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress(Utils.scala:782)If you have problem with mesos master, try mesos://127.0.0.1 instead of mesos://127.0.1.1I0103 20:17:22.329269   340 sched.cpp:330] New master detected at master@127.0.1.1:5050I0103 20:17:22.330749   340 sched.cpp:341] No credentials provided. Attempting to register without authenticationW0103 20:17:22.333531   340 sched.cpp:736] Ignoring framework registered message because it was sentfrom 'master@127.0.0.1:5050' instead of the leading master 'master@127.0.1.1:5050'W0103 20:17:24.040252   339 sched.cpp:736] Ignoring framework registered message because it was sentfrom 'master@127.0.0.1:5050' instead of the leading master 'master@127.0.1.1:5050'W0103 20:17:26.150250   339 sched.cpp:736] Ignoring framework registered message because it was sentfrom 'master@127.0.0.1:5050' instead of the leading master 'master@127.0.1.1:5050'W0103 20:17:26.737604   339 sched.cpp:736] Ignoring framework registered message because it was sentfrom 'master@127.0.0.1:5050' instead of the leading master 'master@127.0.1.1:5050'W0103 20:17:35.241714   336 sched.cpp:736] Ignoring framework registered message because it was sentfrom 'master@127.0.0.1:5050' instead of the leading master 'master@127.0.1.1:5050'",
      "url": " /install/spark_cluster_mode.html",
      "group": "install",
      "excerpt": "This document will guide you how you can build and configure the environment on 3 types of Spark cluster manager(Standalone, Hadoop Yarn, Apache Mesos) with Apache Zeppelin using docker scripts."
    }
    ,
    
  

    "/install/upgrade.html": {
      "title": "Manual Zeppelin version upgrade procedure",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Manual upgrade procedure for ZeppelinBasically, newer version of Zeppelin works with previous version notebook directory and configurations.So, copying notebook and conf directory should be enough.InstructionsStop Zeppelinbin/zeppelin-daemon.sh stopCopy your notebook and conf directory into a backup directoryDownload newer version of Zeppelin and Install. See Install page.Copy backup notebook and conf directory into newer version of Zeppelin notebook and conf directoryStart Zeppelinbin/zeppelin-daemon.sh startMigration GuideUpgrading from Zeppelin 0.6 to 0.7From 0.7, we don't use ZEPPELIN_JAVA_OPTS as default value of ZEPPELIN_INTP_JAVA_OPTS and also the same for ZEPPELIN_MEM/ZEPPELIN_INTP_MEM. If user want to configure the jvm opts of interpreter process, please set ZEPPELIN_INTP_JAVA_OPTS and ZEPPELIN_INTP_MEM explicitly. If you don't set ZEPPELIN_INTP_MEM, Zeppelin will set it to -Xms1024m -Xmx1024m -XX:MaxPermSize=512m by default.Mapping from %jdbc(prefix) to %prefix is no longer available. Instead, you can use %[interpreter alias] with multiple interpreter setttings on GUI.Usage of ZEPPELIN_PORT is not supported in ssl mode. Instead use ZEPPELIN_SSL_PORT to configure the ssl port. Value from ZEPPELIN_PORT is used only when ZEPPELIN_SSL is set to false.The support on Spark 1.1.x to 1.3.x is deprecated.From 0.7, we uses pegdown as the markdown.parser.type option for the %md interpreter. Rendered markdown might be different from what you expectedFrom 0.7 note.json format has been changed to support multiple outputs in a paragraph. Zeppelin will automatically convert old format to new format. 0.6 or lower version can read new note.json format but output will not be displayed. For the detail, see ZEPPELIN-212 and pull request.From 0.7 note storage layer will utilize GitNotebookRepo by default instead of VFSNotebookRepo storage layer, which is an extension of latter one with versioning capabilities on top of it.",
      "url": " /install/upgrade.html",
      "group": "install",
      "excerpt": "This document will guide you through a procedure of manual upgrade your Apache Zeppelin instance to a newer version. Apache Zeppelin keeps backward compatibility for the notebook file format."
    }
    ,
    
  

    "/install/virtual_machine.html": {
      "title": "Apache Zeppelin on Vagrant Virtual Machine",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Zeppelin on Vagrant Virtual MachineOverviewApache Zeppelin distribution includes a script directoryscripts/vagrant/zeppelin-devThis script creates a virtual machine that launches a repeatable, known set of core dependencies required for developing Zeppelin. It can also be used to run an existing Zeppelin build if you don't plan to build from source.For PySpark users, this script includes several helpful Python Libraries.For SparkR users, this script includes several helpful R Libraries.PrerequisitesThis script requires three applications, Ansible, Vagrant and Virtual Box.  All of these applications are freely available as Open Source projects and extremely easy to set up on most operating systems.Create a Zeppelin Ready VMIf you are running Windows and don't yet have python installed, install Python 2.7.x first.Download and Install Vagrant:  Vagrant DownloadsInstall Ansible:  Ansible Python pip installsudo easy_install pipsudo pip install ansibleansible --versionAfter then, please check whether it reports ansible version 1.9.2 or higher.Install Virtual Box: Virtual Box DownloadsType vagrant up  from within the /scripts/vagrant/zeppelin-dev directoryThats it ! You can now run vagrant ssh and this will place you into the guest machines terminal prompt.If you don't wish to build Zeppelin from scratch, run the z-manager installer script while running in the guest VM:curl -fsSL https://raw.githubusercontent.com/NFLabs/z-manager/master/zeppelin-installer.sh | bashBuilding ZeppelinYou can now git clone git://git.apache.org/zeppelin.gitinto a directory on your host machine, or directly in your virtual machine.Cloning Zeppelin into the /scripts/vagrant/zeppelin-dev directory from the host, will allow the directory to be shared between your host and the guest machine.Cloning the project again may seem counter intuitive, since this script likely originated from the project repository.  Consider copying just the vagrant/zeppelin-dev script from the Zeppelin project as a stand alone directory, then once again clone the specific branch you wish to build.Synced folders enable Vagrant to sync a folder on the host machine to the guest machine, allowing you to continue working on your project's files on your host machine, but use the resources in the guest machine to compile or run your project. (1) Synced Folder Description from Vagrant UpBy default, Vagrant will share your project directory (the directory with the Vagrantfile) to /vagrant.  Which means you should be able to build within the guest machine after youcd /vagrant/zeppelinWhat's in this VM?Running the following commands in the guest machine should display these expected versions:node --version should report v0.12.7mvn --version should report Apache Maven 3.3.9 and Java version: 1.7.0_85The virtual machine consists of:Ubuntu Server 14.04 LTSNode.js 0.12.7npm 2.11.3ruby 1.9.3 + rake, make and bundler (only required if building jekyll documentation)Maven 3.3.9GitUnziplibfontconfig to avoid phatomJs missing dependency issuesopenjdk-7-jdkPython addons: pip, matplotlib, scipy, numpy, pandasR and R Packages required to run the R Interpreter and the related R tutorial notebook, including:  Knitr, devtools, repr, rCharts, ggplot2, googleVis, mplot, htmltools, base64enc, data.tableHow to build & run ZeppelinThis assumes you've already cloned the project either on the host machine in the zeppelin-dev directory (to be shared with the guest machine) or cloned directly into a directory while running inside the guest machine.  The following build steps will also include Python and R support via PySpark and SparkR:cd /zeppelinmvn clean package -Pspark-1.6 -Ppyspark -Phadoop-2.4 -Psparkr -DskipTests./bin/zeppelin-daemon.sh startOn your host machine browse to http://localhost:8080/If you turned off port forwarding in the Vagrantfile browse to http://192.168.51.52:8080Tweaking the Virtual MachineIf you plan to run this virtual machine along side other Vagrant images, you may wish to bind the virtual machine to a specific IP address, and not use port fowarding from your local host.Comment out the forward_port line, and uncomment the private_network line in Vagrantfile.  The subnet that works best for your local network will vary so adjust 192.168.*.* accordingly.#config.vm.network "forwarded_port", guest: 8080, host: 8080config.vm.network "private_network", ip: "192.168.51.52"vagrant halt followed by vagrant up will restart the guest machine bound to the IP address of 192.168.51.52.This approach usually is typically required if running other virtual machines that discover each other directly by IP address, such as Spark Masters and Slaves as well as Cassandra Nodes, Elasticsearch Nodes, and other Spark data sources.  You may wish to launch nodes in virtual machines with IP addresses in a subnet that works for your local network, such as: 192.168.51.53, 192.168.51.54, 192.168.51.53, etc..ExtrasPython ExtrasWith Zeppelin running, Numpy, SciPy, Pandas and Matplotlib will be available.  Create a pyspark notebook, and try the below code.%pysparkimport numpyimport scipyimport pandasimport matplotlibprint "numpy " + numpy.__version__print "scipy " + scipy.__version__print "pandas " + pandas.__version__print "matplotlib " + matplotlib.__version__To Test plotting using Matplotlib into a rendered %html SVG image, try%pysparkimport matplotlibmatplotlib.use('Agg')   # turn off interactive charting so this works for server side SVG renderingimport matplotlib.pyplot as pltimport numpy as npimport StringIO# clear out any previous plots on this noteplt.clf()def show(p):    img = StringIO.StringIO()    p.savefig(img, format='svg')    img.seek(0)    print "%html <div style='width:600px'>" + img.buf + "</div>"# Example datapeople = ('Tom', 'Dick', 'Harry', 'Slim', 'Jim')y_pos = np.arange(len(people))performance = 3 + 10 * np.random.rand(len(people))error = np.random.rand(len(people))plt.barh(y_pos, performance, xerr=error, align='center', alpha=0.4)plt.yticks(y_pos, people)plt.xlabel('Performance')plt.title('How fast do you want to go today?')show(plt)R ExtrasWith zeppelin running, an R Tutorial notebook will be available.  The R packages required to run the examples and graphs in this tutorial notebook were installed by this virtual machine.The installed R Packages include: Knitr, devtools, repr, rCharts, ggplot2, googleVis, mplot, htmltools, base64enc, data.table",
      "url": " /install/virtual_machine.html",
      "group": "install",
      "excerpt": "Apache Zeppelin provides a script for running a virtual machine for development through Vagrant. The script will create a virtual machine with core dependencies pre-installed, required for developing Apache Zeppelin."
    }
    ,
    
  

    "/install/yarn_install.html": {
      "title": "Install Zeppelin to connect with existing YARN cluster",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->IntroductionThis page describes how to pre-configure a bare metal node, configure Zeppelin and connect it to existing YARN cluster running Hortonworks flavour of Hadoop. It also describes steps to configure Spark interpreter of Zeppelin.Prepare NodeZeppelin user (Optional)This step is optional, however its nice to run Zeppelin under its own user. In case you do not like to use Zeppelin (hope not) the user could be deleted along with all the packages that were installed for Zeppelin, Zeppelin binary itself and associated directories.Create a zeppelin user and switch to zeppelin user or if zeppelin user is already created then login as zeppelin.useradd zeppelinsu - zeppelinwhoamiAssuming a zeppelin user is created then running whoami command must returnzeppelinIts assumed in the rest of the document that zeppelin user is indeed created and below installation instructions are performed as zeppelin user.List of PrerequisitesCentOS 6.x, Mac OSX, Ubuntu 14.XJava 1.7Hadoop clientSparkInternet connection is required.It's assumed that the node has CentOS 6.x installed on it. Although any version of Linux distribution should work fine.Hadoop clientZeppelin can work with multiple versions & distributions of Hadoop. A complete list is available here. This document assumes Hadoop 2.7.x client libraries including configuration files are installed on Zeppelin node. It also assumes /etc/hadoop/conf contains various Hadoop configuration files. The location of Hadoop configuration files may vary, hence use appropriate location.hadoop versionHadoop 2.7.1.2.3.1.0-2574Subversion git@github.com:hortonworks/hadoop.git -r f66cf95e2e9367a74b0ec88b2df33458b6cff2d0Compiled by jenkins on 2015-07-25T22:36ZCompiled with protoc 2.5.0From source with checksum 54f9bbb4492f92975e84e390599b881dThis command was run using /usr/hdp/2.3.1.0-2574/hadoop/lib/hadoop-common-2.7.1.2.3.1.0-2574.jarSparkSpark is supported out of the box and to take advantage of this, you need to Download appropriate version of Spark binary packages from Spark Download page and unzip it.Zeppelin can work with multiple versions of Spark. A complete list is available here.This document assumes Spark 1.6.0 is installed at /usr/lib/spark.Note: Spark should be installed on the same node as Zeppelin.Note: Spark's pre-built package for CDH 4 doesn't support yarn.ZeppelinCheckout source code from git://git.apache.org/zeppelin.git or download binary package from Download page.You can refer Install page for the details.This document assumes that Zeppelin is located under /home/zeppelin/zeppelin.Zeppelin ConfigurationZeppelin configuration needs to be modified to connect to YARN cluster. Create a copy of zeppelin environment shell script.cp /home/zeppelin/zeppelin/conf/zeppelin-env.sh.template /home/zeppelin/zeppelin/conf/zeppelin-env.shSet the following propertiesexport JAVA_HOME="/usr/java/jdk1.7.0_79"export HADOOP_CONF_DIR="/etc/hadoop/conf"export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"export SPARK_HOME="/usr/lib/spark"As /etc/hadoop/conf contains various configurations of YARN cluster, Zeppelin can now submit Spark/Hive jobs on YARN cluster form its web interface. The value of hdp.version is set to 2.3.1.0-2574. This can be obtained by running the following commandhdp-select status hadoop-client | sed 's/hadoop-client - (.*)/1/'# It returned  2.3.1.0-2574Start/StopStart Zeppelincd /home/zeppelin/zeppelinbin/zeppelin-daemon.sh startAfter successful start, visit http://[zeppelin-server-host-name]:8080 with your web browser.Stop Zeppelinbin/zeppelin-daemon.sh stopInterpreterZeppelin provides various distributed processing frameworks to process data that ranges from Spark, JDBC, Ignite and Lens to name a few. This document describes to configure JDBC & Spark interpreters.HiveZeppelin supports Hive through JDBC interpreter. You might need the information to use Hive and can find in your hive-site.xmlOnce Zeppelin server has started successfully, visit http://[zeppelin-server-host-name]:8080 with your web browser. Click on Interpreter tab next to Notebook dropdown. Look for Hive configurations and set them appropriately. Set them as per Hive installation on YARN cluster.Click on Save button. Once these configurations are updated, Zeppelin will prompt you to restart the interpreter. Accept the prompt and the interpreter will reload the configurations.SparkIt was assumed that 1.6.0 version of Spark is installed at /usr/lib/spark. Look for Spark configurations and click edit button to add the following properties      Property Name    Property Value    Remarks        master    yarn-client    In yarn-client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN.        spark.driver.extraJavaOptions    -Dhdp.version=2.3.1.0-2574            spark.yarn.am.extraJavaOptions    -Dhdp.version=2.3.1.0-2574      Click on Save button. Once these configurations are updated, Zeppelin will prompt you to restart the interpreter. Accept the prompt and the interpreter will reload the configurations.Spark & Hive notebooks can be written with Zeppelin now. The resulting Spark & Hive jobs will run on configured YARN cluster.DebugZeppelin does not emit any kind of error messages on web interface when notebook/paragraph is run. If a paragraph fails it only displays ERROR. The reason for failure needs to be looked into log files which is present in logs directory under zeppelin installation base directory. Zeppelin creates a log file for each kind of interpreter.[zeppelin@zeppelin-3529 logs]$ pwd/home/zeppelin/zeppelin/logs[zeppelin@zeppelin-3529 logs]$ ls -ltotal 844-rw-rw-r-- 1 zeppelin zeppelin  14648 Aug  3 14:45 zeppelin-interpreter-hive-zeppelin-zeppelin-3529.log-rw-rw-r-- 1 zeppelin zeppelin 625050 Aug  3 16:05 zeppelin-interpreter-spark-zeppelin-zeppelin-3529.log-rw-rw-r-- 1 zeppelin zeppelin 200394 Aug  3 21:15 zeppelin-zeppelin-zeppelin-3529.log-rw-rw-r-- 1 zeppelin zeppelin  16162 Aug  3 14:03 zeppelin-zeppelin-zeppelin-3529.out[zeppelin@zeppelin-3529 logs]$",
      "url": " /install/yarn_install.html",
      "group": "install",
      "excerpt": "This page describes how to pre-configure a bare metal node, configure Apache Zeppelin and connect it to existing YARN cluster running Hortonworks flavour of Hadoop."
    }
    ,
    
  

    "/interpreter/alluxio.html": {
      "title": "Alluxio Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Alluxio Interpreter for Apache ZeppelinOverviewAlluxio is a memory-centric distributed storage system enabling reliable data sharing at memory-speed across cluster frameworks.Configuration      Name    Class    Description        alluxio.master.hostname    localhost    Alluxio master hostname        alluxio.master.port    19998    Alluxio master port  Enabling Alluxio InterpreterIn a notebook, to enable the Alluxio interpreter, click on the Gear icon and select Alluxio.Using the Alluxio InterpreterIn a paragraph, use %alluxio to select the Alluxio interpreter and then input all commands.%alluxiohelpTip : Use ( Ctrl + . ) for autocompletion.Interpreter CommandsThe Alluxio interpreter accepts the following commands.            Operation      Syntax      Description              cat      cat "path"      Print the content of the file to the console.              chgrp      chgrp "group" "path"      Change the group of the directory or file.              chmod      chmod "permission" "path"      Change the permission of the directory or file.              chown      chown "owner" "path"      Change the owner of the directory or file.              copyFromLocal      copyFromLocal "source path" "remote path"      Copy the specified file specified by "source path" to the path specified by "remote path".      This command will fail if "remote path" already exists.              copyToLocal      copyToLocal "remote path" "local path"      Copy the specified file from the path specified by "remote path" to a local destination.              count      count "path"      Display the number of folders and files matching the specified prefix in "path".              du      du "path"      Display the size of a file or a directory specified by the input path.              fileInfo      fileInfo "path"      Print the information of the blocks of a specified file.              free      free "path"      Free a file or all files under a directory from Alluxio. If the file/directory is also      in under storage, it will still be available there.              getCapacityBytes      getCapacityBytes      Get the capacity of the AlluxioFS.              getUsedBytes      getUsedBytes      Get number of bytes used in the AlluxioFS.              load      load "path"      Load the data of a file or a directory from under storage into Alluxio.              loadMetadata      loadMetadata "path"      Load the metadata of a file or a directory from under storage into Alluxio.              location      location "path"      Display a list of hosts that have the file data.              ls      ls "path"      List all the files and directories directly under the given path with information such as      size.              mkdir      mkdir "path1" ... "pathn"      Create directory(ies) under the given paths, along with any necessary parent directories.      Multiple paths separated by spaces or tabs. This command will fail if any of the given paths      already exist.              mount      mount "path" "uri"      Mount the underlying file system path "uri" into the Alluxio namespace as "path". The "path"      is assumed not to exist and is created by the operation. No data or metadata is loaded from under      storage into Alluxio. After a path is mounted, operations on objects under the mounted path are      mirror to the mounted under storage.              mv      mv "source" "destination"      Move a file or directory specified by "source" to a new location "destination". This command      will fail if "destination" already exists.              persist      persist "path"      Persist a file or directory currently stored only in Alluxio to the underlying file system.              pin      pin "path"      Pin the given file to avoid evicting it from memory. If the given path is a directory, it      recursively pins all the files contained and any new files created within this directory.              report      report "path"      Report to the master that a file is lost.              rm      rm "path"      Remove a file. This command will fail if the given path is a directory rather than a file.              setTtl      setTtl "time"      Set the TTL (time to live) in milliseconds to a file.              tail      tail "path"      Print the last 1KB of the specified file to the console.              touch      touch "path"      Create a 0-byte file at the specified location.              unmount      unmount "path"      Unmount the underlying file system path mounted in the Alluxio namespace as "path". Alluxio      objects under "path" are removed from Alluxio, but they still exist in the previously mounted      under storage.              unpin      unpin "path"      Unpin the given file to allow Alluxio to evict this file again. If the given path is a      directory, it recursively unpins all files contained and any new files created within this      directory.              unsetTtl      unsetTtl      Remove the TTL (time to live) setting from a file.      How to test it's workingBe sure to have configured correctly the Alluxio interpreter, then open a new paragraph and type one of the above commands.Below a simple example to show how to interact with Alluxio interpreter.Following steps are performed:using sh interpreter a new text file is created on local machineusing Alluxio interpreter:is listed the content of the afs (Alluxio File System) rootthe file previously created is copied to afsis listed again the content of the afs root to check the existence of the new copied fileis showed the content of the copied file (using the tail command)the file previously copied to afs is copied to local machine using sh interpreter it's checked the existence of the new file copied from Alluxio and its content is showed  ",
      "url": " /interpreter/alluxio.html",
      "group": "interpreter",
      "excerpt": "Alluxio is a memory-centric distributed storage system enabling reliable data sharing at memory-speed across cluster frameworks."
    }
    ,
    
  

    "/interpreter/beam.html": {
      "title": "Beam interpreter in Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Beam interpreter for Apache ZeppelinOverviewApache Beam is an open source unified platform for data processing pipelines. A pipeline can be build using one of the Beam SDKs.The execution of the pipeline is done by different Runners. Currently, Beam supports Apache Flink Runner, Apache Spark Runner, and Google Dataflow Runner.How to useBasically, you can write normal Beam java code where you can determine the Runner. You should write the main method inside a class becuase the interpreter invoke this main to execute the pipeline. Unlike Zeppelin normal pattern, each paragraph is considered as a separate job, there isn't any relation to any other paragraph.The following is a demonstration of a word count example with data represented in array of stringsBut it can read data from files by replacing Create.of(SENTENCES).withCoder(StringUtf8Coder.of()) with TextIO.Read.from("path/to/filename.txt")%beam// most used importsimport org.apache.beam.sdk.coders.StringUtf8Coder;import org.apache.beam.sdk.transforms.Create;import java.io.Serializable;import java.util.Arrays;import java.util.List;import java.util.ArrayList;import org.apache.spark.api.java.*;import org.apache.spark.api.java.function.Function;import org.apache.spark.SparkConf;import org.apache.spark.streaming.*;import org.apache.spark.SparkContext;import org.apache.beam.runners.direct.*;import org.apache.beam.sdk.runners.*;import org.apache.beam.sdk.options.*;import org.apache.beam.runners.spark.*;import org.apache.beam.runners.spark.io.ConsoleIO;import org.apache.beam.runners.flink.*;import org.apache.beam.runners.flink.examples.WordCount.Options;import org.apache.beam.sdk.Pipeline;import org.apache.beam.sdk.io.TextIO;import org.apache.beam.sdk.options.PipelineOptionsFactory;import org.apache.beam.sdk.transforms.Count;import org.apache.beam.sdk.transforms.DoFn;import org.apache.beam.sdk.transforms.MapElements;import org.apache.beam.sdk.transforms.ParDo;import org.apache.beam.sdk.transforms.SimpleFunction;import org.apache.beam.sdk.values.KV;import org.apache.beam.sdk.options.PipelineOptions;public class MinimalWordCount {  static List<String> s = new ArrayList<>();  static final String[] SENTENCES_ARRAY = new String[] {    "Hadoop is the Elephant King!",    "A yellow and elegant thing.",    "He never forgets",    "Useful data, or lets",    "An extraneous element cling!",    "A wonderful king is Hadoop.",    "The elephant plays well with Sqoop.",    "But what helps him to thrive",    "Are Impala, and Hive,",    "And HDFS in the group.",    "Hadoop is an elegant fellow.",    "An elephant gentle and mellow.",    "He never gets mad,",    "Or does anything bad,",    "Because, at his core, he is yellow",    };    static final List<String> SENTENCES = Arrays.asList(SENTENCES_ARRAY);  public static void main(String[] args) {    Options options = PipelineOptionsFactory.create().as(Options.class);    options.setRunner(FlinkRunner.class);    Pipeline p = Pipeline.create(options);    p.apply(Create.of(SENTENCES).withCoder(StringUtf8Coder.of()))         .apply("ExtractWords", ParDo.of(new DoFn<String, String>() {           @Override           public void processElement(ProcessContext c) {             for (String word : c.element().split("[^a-zA-Z']+")) {               if (!word.isEmpty()) {                 c.output(word);               }             }           }         }))        .apply(Count.<String> perElement())        .apply("FormatResults", ParDo.of(new DoFn<KV<String, Long>, String>() {          @Override          public void processElement(DoFn<KV<String, Long>, String>.ProcessContext arg0)            throws Exception {            s.add("n" + arg0.element().getKey() + "t" + arg0.element().getValue());            }        }));    p.run();    System.out.println("%table wordtcount");    for (int i = 0; i < s.size(); i++) {      System.out.print(s.get(i));    }  }}",
      "url": " /interpreter/beam.html",
      "group": "interpreter",
      "excerpt": "Apache Beam is an open source, unified programming model that you can use to create a data processing pipeline."
    }
    ,
    
  

    "/interpreter/bigquery.html": {
      "title": "BigQuery Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->BigQuery Interpreter for Apache ZeppelinOverviewBigQuery is a highly scalable no-ops data warehouse in the Google Cloud Platform. Querying massive datasets can be time consuming and expensive without the right hardware and infrastructure. Google BigQuery solves this problem by enabling super-fast SQL queries against append-only tables using the processing power of Google's infrastructure. Simply move your data into BigQuery and let us handle the hard work. You can control access to both the project and your data based on your business needs, such as giving others the ability to view or query your data.  Configuration      Name    Default Value    Description        zeppelin.bigquery.project_id          Google Project Id        zeppelin.bigquery.wait_time    5000    Query Timeout in Milliseconds        zeppelin.bigquery.max_no_of_rows    100000    Max result set size  BigQuery APIZeppelin is built against BigQuery API version v2-rev265-1.21.0 - API JavadocsEnabling the BigQuery InterpreterIn a notebook, to enable the BigQuery interpreter, click the Gear icon and select bigquery.Setup service account credentialsIn order to run BigQuery interpreter outside of Google Cloud Engine you need to provide authentication credentials,by following this instructions:Go to the API Console Credentials pageFrom the project drop-down, select your project.On the Credentials page, select the Create credentials drop-down, then select Service account key.From the Service account drop-down, select an existing service account or create a new one.For Key type, select the JSON key option, then select Create. The file automatically downloads to your computer.Put the *.json file you just downloaded in a directory of your choosing. This directory must be private (you can't let anyone get access to this), but accessible to your Zeppelin instance.Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the JSON file downloaded.either though GUI: in interpreter configuration page property names in CAPITAL_CASE set up env varsor though zeppelin-env.sh: just add it to the end of the file.Using the BigQuery InterpreterIn a paragraph, use %bigquery.sql to select the BigQuery interpreter and then input SQL statements against your datasets stored in BigQuery.You can use BigQuery SQL Reference to build your own SQL.For Example, SQL to query for top 10 departure delays across airports using the flights public dataset%bigquery.sqlSELECT departure_airport,count(case when departure_delay>0 then 1 else 0 end) as no_of_delays FROM [bigquery-samples:airline_ontime_data.flights] group by departure_airport order by 2 desc limit 10Another Example, SQL to query for most commonly used java packages from the github data hosted in BigQuery %bigquery.sqlSELECT  package,  COUNT(*) countFROM (  SELECT    REGEXP_EXTRACT(line, r' ([a-z0-9._]*).') package,    id  FROM (    SELECT      SPLIT(content, 'n') line,      id    FROM      [bigquery-public-data:github_repos.sample_contents]    WHERE      content CONTAINS 'import'      AND sample_path LIKE '%.java'    HAVING      LEFT(line, 6)='import' )  GROUP BY    package,    id )GROUP BY  1ORDER BY  count DESCLIMIT  40Technical descriptionFor in-depth technical details on current implementation please refer to bigquery/README.md.",
      "url": " /interpreter/bigquery.html",
      "group": "interpreter",
      "excerpt": "BigQuery is a highly scalable no-ops data warehouse in the Google Cloud Platform."
    }
    ,
    
  

    "/interpreter/cassandra.html": {
      "title": "Cassandra CQL Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Cassandra CQL Interpreter for Apache Zeppelin      Name    Class    Description        %cassandra    CassandraInterpreter    Provides interpreter for Apache Cassandra CQL query language  Enabling Cassandra InterpreterIn a notebook, to enable the Cassandra interpreter, click on the Gear icon and select Cassandra  Using the Cassandra InterpreterIn a paragraph, use %cassandra to select the Cassandra interpreter and then input all commands.To access the interactive help, type HELP;    Interpreter CommandsThe Cassandra interpreter accepts the following commands            Command Type      Command Name      Description              Help command      HELP      Display the interactive help menu              Schema commands      DESCRIBE KEYSPACE, DESCRIBE CLUSTER, DESCRIBE TABLES ...      Custom commands to describe the Cassandra schema              Option commands      @consistency, @retryPolicy, @fetchSize ...      Inject runtime options to all statements in the paragraph              Prepared statement commands      @prepare, @bind, @remove_prepared      Let you register a prepared command and re-use it later by injecting bound values              Native CQL statements      All CQL-compatible statements (SELECT, INSERT, CREATE ...)      All CQL statements are executed directly against the Cassandra server      CQL statementsThis interpreter is compatible with any CQL statement supported by Cassandra. Ex:INSERT INTO users(login,name) VALUES('jdoe','John DOE');SELECT * FROM users WHERE login='jdoe';Each statement should be separated by a semi-colon ( ; ) except the special commands below:@prepare@bind@remove_prepare@consistency@serialConsistency@timestamp@retryPolicy@fetchSize@requestTimeOutMulti-line statements as well as multiple statements on the same line are also supported as long as they are separated by a semi-colon. Ex:USE spark_demo;SELECT * FROM albums_by_country LIMIT 1; SELECT * FROM countries LIMIT 1;SELECT *FROM artistsWHERE login='jlennon';Batch statements are supported and can span multiple lines, as well as DDL(CREATE/ALTER/DROP) statements:BEGIN BATCH    INSERT INTO users(login,name) VALUES('jdoe','John DOE');    INSERT INTO users_preferences(login,account_type) VALUES('jdoe','BASIC');APPLY BATCH;CREATE TABLE IF NOT EXISTS test(    key int PRIMARY KEY,    value text);CQL statements are case-insensitive (except for column names and values). This means that the following statements are equivalent and valid:INSERT INTO users(login,name) VALUES('jdoe','John DOE');Insert into users(login,name) vAlues('hsue','Helen SUE');The complete list of all CQL statements and versions can be found below:         Cassandra Version     Documentation Link           3.x             <a target="_blank"          href="http://docs.datastax.com/en/cql/3.3/cql/cqlIntro.html">          http://docs.datastax.com/en/cql/3.3/cql/cqlIntro.html                        2.2             <a target="_blank"          href="http://docs.datastax.com/en/cql/3.3/cql/cqlIntro.html">          http://docs.datastax.com/en/cql/3.3/cql/cqlIntro.html                        2.1 & 2.0             <a target="_blank"          href="http://docs.datastax.com/en/cql/3.1/cql/cql_intro_c.html">          http://docs.datastax.com/en/cql/3.1/cql/cqlintroc.html                        1.2             <a target="_blank"          href="http://docs.datastax.com/en/cql/3.0/cql/aboutCQL.html">          http://docs.datastax.com/en/cql/3.0/cql/aboutCQL.html                 Comments in statementsIt is possible to add comments between statements. Single line comments start with the hash sign (#) or double slashes (//). Multi-line comments are enclosed between /** and **/. Ex:#Single line comment style 1INSERT INTO users(login,name) VALUES('jdoe','John DOE');//Single line comment style 2/** Multi line comments **/Insert into users(login,name) vAlues('hsue','Helen SUE');Syntax ValidationThe interpreters is shipped with a built-in syntax validator. This validator only checks for basic syntax errors.All CQL-related syntax validation is delegated directly to CassandraMost of the time, syntax errors are due to missing semi-colons between statements or typo errors.Schema commandsTo make schema discovery easier and more interactive, the following commands are supported:         Command     Description           DESCRIBE CLUSTER;     Show the current cluster name and its partitioner           DESCRIBE KEYSPACES;     List all existing keyspaces in the cluster and their configuration (replication factor, durable write ...)           DESCRIBE TABLES;     List all existing keyspaces in the cluster and for each, all the tables name           DESCRIBE TYPES;     List all existing keyspaces in the cluster and for each, all the user-defined types name           DESCRIBE FUNCTIONS;     List all existing keyspaces in the cluster and for each, all the functions name           DESCRIBE AGGREGATES;     List all existing keyspaces in the cluster and for each, all the aggregates name           DESCRIBE MATERIALIZED VIEWS;     List all existing keyspaces in the cluster and for each, all the materialized views name           DESCRIBE KEYSPACE <keyspacename>;     Describe the given keyspace configuration and all its table details (name, columns, ...)           DESCRIBE TABLE (<keyspacename>).<tablename>;             Describe the given table. If the keyspace is not provided, the current logged in keyspace is used.        If there is no logged in keyspace, the default system keyspace is used.        If no table is found, an error message is raised                DESCRIBE TYPE (<keyspacename>).<typename>;             Describe the given type(UDT). If the keyspace is not provided, the current logged in keyspace is used.        If there is no logged in keyspace, the default system keyspace is used.        If no type is found, an error message is raised                DESCRIBE FUNCTION (<keyspacename>).<functionname>;     Describe the given function. If the keyspace is not provided, the current logged in keyspace is used.         If there is no logged in keyspace, the default system keyspace is used.         If no function is found, an error message is raised                DESCRIBE AGGREGATE (<keyspacename>).<aggregatename>;     Describe the given aggregate. If the keyspace is not provided, the current logged in keyspace is used.         If there is no logged in keyspace, the default system keyspace is used.         If no aggregate is found, an error message is raised                DESCRIBE MATERIALIZED VIEW (<keyspacename>).<view_name>;     Describe the given view. If the keyspace is not provided, the current logged in keyspace is used.         If there is no logged in keyspace, the default system keyspace is used.         If no view is found, an error message is raised         The schema objects (cluster, keyspace, table, type, function and aggregate) are displayed in a tabular format.There is a drop-down menu on the top left corner to expand objects details. On the top right menu is shown the Icon legend.  Runtime ParametersSometimes you want to be able to pass runtime query parameters to your statements.Those parameters are not part of the CQL specs and are specific to the interpreter.Below is the list of all parameters:         Parameter     Syntax     Description           Consistency Level     @consistency=value     Apply the given consistency level to all queries in the paragraph           Serial Consistency Level     @serialConsistency=value     Apply the given serial consistency level to all queries in the paragraph           Timestamp     @timestamp=long value             Apply the given timestamp to all queries in the paragraph.        Please note that timestamp value passed directly in CQL statement will override this value                 Retry Policy     @retryPolicy=value     Apply the given retry policy to all queries in the paragraph           Fetch Size     @fetchSize=integer value     Apply the given fetch size to all queries in the paragraph           Request Time Out     @requestTimeOut=integer value     Apply the given request timeout in millisecs to all queries in the paragraph    Some parameters only accept restricted values:         Parameter     Possible Values           Consistency Level     ALL, ANY, ONE, TWO, THREE, QUORUM, LOCALONE, LOCALQUORUM, EACHQUORUM           Serial Consistency Level     SERIAL, LOCALSERIAL           Timestamp     Any long value           Retry Policy     DEFAULT, DOWNGRADINGCONSISTENCY, FALLTHROUGH, LOGGINGDEFAULT, LOGGINGDOWNGRADING, LOGGINGFALLTHROUGH           Fetch Size     Any integer value    Please note that you should not add semi-colon ( ; ) at the end of each parameter statementSome examples:CREATE TABLE IF NOT EXISTS spark_demo.ts(    key int PRIMARY KEY,    value text);TRUNCATE spark_demo.ts;// Timestamp in the past@timestamp=10// Force timestamp directly in the first insertINSERT INTO spark_demo.ts(key,value) VALUES(1,'first insert') USING TIMESTAMP 100;// Select some data to make the clock turnSELECT * FROM spark_demo.albums LIMIT 100;// Now insert using the timestamp parameter set at the beginning(10)INSERT INTO spark_demo.ts(key,value) VALUES(1,'second insert');// Check for the result. You should see 'first insert'SELECT value FROM spark_demo.ts WHERE key=1;Some remarks about query parameters:many query parameters can be set in the same paragraphif the same query parameter is set many time with different values, the interpreter only take into account the first valueeach query parameter applies to all CQL statements in the same paragraph, unless you override the option using plain CQL text (like forcing timestamp with the USING clause)the order of each query parameter with regard to CQL statement does not matterSupport for Prepared StatementsFor performance reason, it is better to prepare statements before-hand and reuse them later by providing bound values.This interpreter provides 3 commands to handle prepared and bound statements:@prepare@bind@remove_preparedExample:@prepare[statement-name]=...@bind[statement-name]=’text’, 1223, ’2015-07-30 12:00:01’, null, true, [‘list_item1’, ’list_item2’]@bind[statement-name-with-no-bound-value]@remove_prepare[statement-name]@prepareYou can use the syntax "@prepare[statement-name]=SELECT..." to create a prepared statement.The statement-name is mandatory because the interpreter prepares the given statement with the Java driver andsaves the generated prepared statement in an internal hash map, using the provided statement-name as search key.Please note that this internal prepared statement map is shared with all notebooks and all paragraphs becausethere is only one instance of the interpreter for CassandraIf the interpreter encounters many @prepare for the same statement-name (key), only the first statement will be taken into account.Example:@prepare[select]=SELECT * FROM spark_demo.albums LIMIT ?@prepare[select]=SELECT * FROM spark_demo.artists LIMIT ?For the above example, the prepared statement is SELECT * FROM spark_demo.albums LIMIT ?.`SELECT * FROM spark_demo.artists LIMIT ? is ignored because an entry already exists in the prepared statements map with the key select.In the context of Zeppelin, a notebook can be scheduled to be executed at regular interval,thus it is necessary to avoid re-preparing many time the same statement (considered an anti-pattern).@bindOnce the statement is prepared (possibly in a separated notebook/paragraph). You can bind values to it:@bind[select_first]=10Bound values are not mandatory for the @bind statement. However if you provide bound values, they need to comply to some syntax:String values should be enclosed between simple quotes ( ‘ )Date values should be enclosed between simple quotes ( ‘ ) and respect the formats:yyyy-MM-dd HH:MM:ssyyyy-MM-dd HH:MM:ss.SSSnull is parsed as-isboolean (true|false) are parsed as-iscollection values must follow the standard CQL syntax:list: [‘listitem1’, ’listitem2’, ...]set: {‘setitem1’, ‘setitem2’, …}map: {‘key1’: ‘val1’, ‘key2’: ‘val2’, …}tuple values should be enclosed between parenthesis (see Tuple CQL syntax): (‘text’, 123, true)udt values should be enclosed between brackets (see UDT CQL syntax): {streename: ‘Beverly Hills’, number: 104, zipcode: 90020, state: ‘California’, …}It is possible to use the @bind statement inside a batch:BEGIN BATCH   @bind[insert_user]='jdoe','John DOE'   UPDATE users SET age = 27 WHERE login='hsue';APPLY BATCH;@remove_prepareTo avoid for a prepared statement to stay forever in the prepared statement map, you can use the@remove_prepare[statement-name] syntax to remove it.Removing a non-existing prepared statement yields no error.Using Dynamic FormsInstead of hard-coding your CQL queries, it is possible to use Zeppelin dynamic form syntax to inject simple value or multiple choices forms.The legacy mustache syntax ( {{ }} ) to bind input text and select form is still supported but is deprecated and will be removed in future releases.LegacyThe syntax for simple parameter is: {{input_Label=default value}}. The default value is mandatory because the first time the paragraph is executed,we launch the CQL query before rendering the form so at least one value should be provided.The syntax for multiple choices parameter is: {{input_Label=value1 | value2 | … | valueN }}. By default the first choice is used for CQL querythe first time the paragraph is executed.Example:#Secondary index on performer styleSELECT name, country, performerFROM spark_demo.performersWHERE name='${performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}'AND styles CONTAINS '${style=Rock}';In the above example, the first CQL query will be executed for performer='Sheryl Crow' AND style='Rock'.For subsequent queries, you can change the value directly using the form.Please note that we enclosed the ${ } block between simple quotes ( ' ) because Cassandra expects a String here.We could have also use the ${style='Rock'} syntax but this time, the value displayed on the form is 'Rock' and not Rock.It is also possible to use dynamic forms for prepared statements:@bind[select]=='${performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}', '${style=Rock}'Shared statesIt is possible to execute many paragraphs in parallel. However, at the back-end side, we’re still using synchronous queries.Asynchronous execution is only possible when it is possible to return a Future value in the InterpreterResult.It may be an interesting proposal for the Zeppelin project.Recently, Zeppelin allows you to choose the level of isolation for your interpreters (see Interpreter Binding Mode ).Long story short, you have 3 available bindings:shared : same JVM and same Interpreter instance for all notesscoped : same JVM but different Interpreter instances, one for each noteisolated: different JVM running a single Interpreter instance, one JVM for each noteUsing the shared binding, the same com.datastax.driver.core.Session object is used for all notes and paragraphs.Consequently, if you use the USE keyspace name; statement to log into a keyspace, it will change the keyspace forall current users of the Cassandra interpreter because we only create 1 com.datastax.driver.core.Session objectper instance of Cassandra interpreter.The same remark does apply to the prepared statement hash map, it is shared by all users using the same instance of Cassandra interpreter.When using scoped binding, in the same JVM Zeppelin will create multiple instances of the Cassandra interpreter, thus multiple com.datastax.driver.core.Session objects. Beware of resource and memory usage using this binding ! The isolated mode is the most extreme and will create as many JVM/com.datastax.driver.core.Session object as there are distinct notes.Interpreter ConfigurationTo configure the Cassandra interpreter, go to the Interpreter menu and scroll down to change the parameters.The Cassandra interpreter is using the official Cassandra Java Driver and most of the parameters are usedto configure the Java driverBelow are the configuration parameters and their default value.        Property Name     Description     Default Value           cassandra.cluster     Name of the Cassandra cluster to connect to     Test Cluster           cassandra.compression.protocol     On wire compression. Possible values are: NONE, SNAPPY, LZ4     NONE           cassandra.credentials.username     If security is enable, provide the login     none           cassandra.credentials.password     If security is enable, provide the password     none           cassandra.hosts             Comma separated Cassandra hosts (DNS name or IP address).                Ex: '192.168.0.12,node2,node3'           localhost           cassandra.interpreter.parallelism     Number of concurrent paragraphs(queries block) that can be executed     10           cassandra.keyspace             Default keyspace to connect to.                  It is strongly recommended to let the default value          and prefix the table name with the actual keyspace          in all of your queries                  system           cassandra.load.balancing.policy             Load balancing policy. Default = new TokenAwarePolicy(new DCAwareRoundRobinPolicy())        To Specify your own policy, provide the fully qualify class name (FQCN) of your policy.        At runtime the interpreter will instantiate the policy using        Class.forName(FQCN)          DEFAULT           cassandra.max.schema.agreement.wait.second     Cassandra max schema agreement wait in second     10           cassandra.pooling.core.connection.per.host.local     Protocol V2 and below default = 2. Protocol V3 and above default = 1     2           cassandra.pooling.core.connection.per.host.remote     Protocol V2 and below default = 1. Protocol V3 and above default = 1     1           cassandra.pooling.heartbeat.interval.seconds     Cassandra pool heartbeat interval in secs     30           cassandra.pooling.idle.timeout.seconds     Cassandra idle time out in seconds     120           cassandra.pooling.max.connection.per.host.local     Protocol V2 and below default = 8. Protocol V3 and above default = 1     8           cassandra.pooling.max.connection.per.host.remote     Protocol V2 and below default = 2. Protocol V3 and above default = 1     2           cassandra.pooling.max.request.per.connection.local     Protocol V2 and below default = 128. Protocol V3 and above default = 1024     128           cassandra.pooling.max.request.per.connection.remote     Protocol V2 and below default = 128. Protocol V3 and above default = 256     128           cassandra.pooling.new.connection.threshold.local     Protocol V2 and below default = 100. Protocol V3 and above default = 800     100           cassandra.pooling.new.connection.threshold.remote     Protocol V2 and below default = 100. Protocol V3 and above default = 200     100           cassandra.pooling.pool.timeout.millisecs     Cassandra pool time out in millisecs     5000           cassandra.protocol.version     Cassandra binary protocol version     4           cassandra.query.default.consistency           Cassandra query default consistency level            Available values: ONE, TWO, THREE, QUORUM, LOCALONE, LOCALQUORUM, EACHQUORUM, ALL          ONE           cassandra.query.default.fetchSize     Cassandra query default fetch size     5000           cassandra.query.default.serial.consistency           Cassandra query default serial consistency level            Available values: SERIAL, LOCALSERIAL          SERIAL           cassandra.reconnection.policy             Cassandra Reconnection Policy.        Default = new ExponentialReconnectionPolicy(1000, 10 * 60 * 1000)        To Specify your own policy, provide the fully qualify class name (FQCN) of your policy.        At runtime the interpreter will instantiate the policy using        Class.forName(FQCN)          DEFAULT           cassandra.retry.policy             Cassandra Retry Policy.        Default = DefaultRetryPolicy.INSTANCE        To Specify your own policy, provide the fully qualify class name (FQCN) of your policy.        At runtime the interpreter will instantiate the policy using        Class.forName(FQCN)          DEFAULT           cassandra.socket.connection.timeout.millisecs     Cassandra socket default connection timeout in millisecs     500           cassandra.socket.read.timeout.millisecs     Cassandra socket read timeout in millisecs     12000           cassandra.socket.tcp.no_delay     Cassandra socket TCP no delay     true           cassandra.speculative.execution.policy             Cassandra Speculative Execution Policy.        Default = NoSpeculativeExecutionPolicy.INSTANCE        To Specify your own policy, provide the fully qualify class name (FQCN) of your policy.        At runtime the interpreter will instantiate the policy using        Class.forName(FQCN)          DEFAULT    Change Log3.0 (Zeppelin 0.7.2) :Update documentationUpdate interactive documentationAdd support for binary protocol V4Implement new @requestTimeOut runtime optionUpgrade Java driver version to 3.0.1Allow interpreter to add dynamic forms programmatically when using FormType.SIMPLEAllow dynamic form using default Zeppelin syntaxFixing typo on FallThroughPolicyLook for data in AngularObjectRegistry before creating dynamic formAdd missing support for ALTER statements2.0 (Zeppelin 0.7.2) :Update help menu and add changelogAdd Support for User Defined Functions, User Defined Aggregates and Materialized ViewsUpgrade Java driver version to 3.0.0-rc11.0 (Zeppelin 0.5.5-incubating) :Initial versionBugs & ContactsIf you encounter a bug for this interpreter, please create a JIRA ticket and ping me on Twitter at @doanduyhai",
      "url": " /interpreter/cassandra.html",
      "group": "interpreter",
      "excerpt": "Apache Cassandra database is the right choice when you need scalability and high availability without compromising performance."
    }
    ,
    
  

    "/interpreter/elasticsearch.html": {
      "title": "Elasticsearch Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Elasticsearch Interpreter for Apache ZeppelinOverviewElasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements.Configuration      Property    Default    Description        elasticsearch.cluster.name    elasticsearch    Cluster name        elasticsearch.host    localhost    Host of a node in the cluster        elasticsearch.port    9300    Connection port ( Important: it depends on the client type, transport or http)        elasticsearch.client.type    transport    The type of client for Elasticsearch (transport or http)( Important: the port depends on this value)        elasticsearch.basicauth.username        Username for a basic authentication (http)        elasticsearch.basicauth.password        Password for a basic authentication (http)        elasticsearch.result.size    10    The size of the result set of a search query    Note #1 : You can add more properties to configure the Elasticsearch client.Note #2 : If you use Shield, you can add a property named shield.user with a value containing the name and the password ( format: username:password ). For more details about Shield configuration, consult the Shield reference guide. Do not forget, to copy the shield client jar in the interpreter directory (ZEPPELIN_HOME/interpreters/elasticsearch).Enabling the Elasticsearch InterpreterIn a notebook, to enable the Elasticsearch interpreter, click the Gear icon and select Elasticsearch.Using the Elasticsearch InterpreterIn a paragraph, use %elasticsearch to select the Elasticsearch interpreter and then input all commands. To get the list of available commands, use help.%elasticsearchhelpElasticsearch interpreter:General format: <command> /<indices>/<types>/<id> <option> <JSON>  - indices: list of indices separated by commas (depends on the command)  - types: list of document types separated by commas (depends on the command)Commands:  - search /indices/types <query>    . indices and types can be omitted (at least, you have to provide '/')    . a query is either a JSON-formatted query, nor a lucene query  - size <value>    . defines the size of the result set (default value is in the config)    . if used, this command must be declared before a search command  - count /indices/types <query>    . same comments as for the search  - get /index/type/id  - delete /index/type/id  - index /index/type/id <json-formatted document>    . the id can be omitted, elasticsearch will generate oneTip : Use ( Ctrl + . ) for autocompletion.GetWith the get command, you can find a document by id. The result is a JSON document.%elasticsearchget /index/type/idExample:SearchWith the search command, you can send a search query to Elasticsearch. There are two formats of query:You can provide a JSON-formatted query, that is exactly what you provide when you use the REST API of Elasticsearch.See Elasticsearch search API reference document for more details about the content of the search queries.You can also provide the content of a query_string.This is a shortcut to a query like that: { "query": { "query_string": { "query": "__HERE YOUR QUERY__", "analyze_wildcard": true } } }See Elasticsearch query string syntax for more details about the content of such a query.%elasticsearchsearch /index1,index2,.../type1,type2,...  <JSON document containing the query or query_string elements>If you want to modify the size of the result set, you can add a line that is setting the size, before your search command.%elasticsearchsize 50search /index1,index2,.../type1,type2,...  <JSON document containing the query or query_string elements>A search query can also contain aggregations. If there is at least one aggregation, the result of the first aggregation is shown, otherwise, you get the search hits.Examples:With a JSON query:%elasticsearchsearch / { "query": { "match_all": { } } }%elasticsearchsearch /logs { "query": { "query_string": { "query": "request.method:GET AND status:200" } } }%elasticsearchsearch /logs { "aggs": {"content_length_stats": {  "extended_stats": {    "field": "content_length"  }}} }With query_string elements:%elasticsearchsearch /logs request.method:GET AND status:200%elasticsearchsearch /logs (404 AND (POST OR DELETE))Important : a document in Elasticsearch is a JSON document, so it is hierarchical, not flat as a row in a SQL table.For the Elastic interpreter, the result of a search query is flattened.Suppose we have a JSON document:{  "date": "2015-12-08T21:03:13.588Z",  "request": {    "method": "GET",    "url": "/zeppelin/4cd001cd-c517-4fa9-b8e5-a06b8f4056c4",    "headers": [ "Accept: *.*", "Host: apache.org"]  },  "status": "403",  "content_length": 1234}The data will be flattened like this:content_lengthdaterequest.headers[0]request.headers[1]request.methodrequest.urlstatus12342015-12-08T21:03:13.588ZAccept: *.*Host: apache.orgGET/zeppelin/4cd001cd-c517-4fa9-b8e5-a06b8f4056c4403Examples:With a table containing the results:You can also use a predefined diagram:With a JSON query:With a JSON query containing a fields parameter (for filtering the fields in the response): in this case, all the fields values in the response are arrays, so, after flattening the result, the format of all the field names is field_name[x]With a query string:With a query containing a multi-value metric aggregation:With a query containing a multi-bucket aggregation:CountWith the count command, you can count documents available in some indices and types. You can also provide a query.%elasticsearchcount /index1,index2,.../type1,type2,... <JSON document containing the query OR a query string>Examples:Without query:With a query:IndexWith the index command, you can insert/update a document in Elasticsearch.%elasticsearchindex /index/type/id <JSON document>%elasticsearchindex /index/type <JSON document>DeleteWith the delete command, you can delete a document.%elasticsearchdelete /index/type/idApply Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form inside your queries. You can use both the text input and select form parameterization features.%elasticsearchsize ${limit=10}search /index/type { "query": { "match_all": { } } }",
      "url": " /interpreter/elasticsearch.html",
      "group": "interpreter",
      "excerpt": "Elasticsearch is a highly scalable open-source full-text search and analytics engine."
    }
    ,
    
  

    "/interpreter/flink.html": {
      "title": "Flink Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Flink interpreter for Apache ZeppelinOverviewApache Flink is an open source platform for distributed stream and batch data processing. Flink’s core is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams. Flink also builds batch processing on top of the streaming engine, overlaying native iteration support, managed memory, and program optimization.How to start local Flink cluster, to test the interpreterZeppelin comes with pre-configured flink-local interpreter, which starts Flink in a local mode on your machine, so you do not need to install anything.How to configure interpreter to point to Flink clusterAt the "Interpreters" menu, you have to create a new Flink interpreter and provide next properties:      property    value    Description        host    local    host name of running JobManager. 'local' runs flink in local mode (default)        port    6123    port of running JobManager  For more information about Flink configuration, you can find it here.How to test it's workingYou can find an example of Flink usage in the Zeppelin Tutorial folder or try the following word count example, by using the Zeppelin notebook from Till Rohrmann's presentation Interactive data analysis with Apache Flink for Apache Flink Meetup.%shrm 10.txt.utf-8wget http://www.gutenberg.org/ebooks/10.txt.utf-8%flinkcase class WordCount(word: String, frequency: Int)val bible:DataSet[String] = benv.readTextFile("10.txt.utf-8")val partialCounts: DataSet[WordCount] = bible.flatMap{    line =>        """bw+b""".r.findAllIn(line).map(word => WordCount(word, 1))//        line.split(" ").map(word => WordCount(word, 1))}val wordCounts = partialCounts.groupBy("word").reduce{    (left, right) => WordCount(left.word, left.frequency + right.frequency)}val result10 = wordCounts.first(10).collect()",
      "url": " /interpreter/flink.html",
      "group": "interpreter",
      "excerpt": "Apache Flink is an open source platform for distributed stream and batch data processing."
    }
    ,
    
  

    "/interpreter/geode.html": {
      "title": "Geode/Gemfire OQL Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Geode/Gemfire OQL Interpreter for Apache ZeppelinOverview      Name    Class    Description        %geode.oql    GeodeOqlInterpreter    Provides OQL environment for Apache Geode  This interpreter supports the Geode Object Query Language (OQL).  With the OQL-based querying language:You can query on any arbitrary objectYou can navigate object collectionsYou can invoke methods and access the behavior of objectsData mapping is supportedYou are not required to declare types. Since you do not need type definitions, you can work across multiple languagesYou are not constrained by a schemaThis Video Tutorial illustrates some of the features provided by the Geode Interpreter.Create InterpreterBy default Zeppelin creates one Geode/OQL instance. You can remove it or create more instances.Multiple Geode instances can be created, each configured to the same or different backend Geode cluster. But over time a  Notebook can have only one Geode interpreter instance bound. That means you cannot connect to different Geode clusters in the same Notebook. This is a known Zeppelin limitation.To create new Geode instance open the Interpreter section and click the +Create button. Pick a Name of your choice and from the Interpreter drop-down select geode.  Then follow the configuration instructions and Save the new instance.Note: The Name of the instance is used only to distinguish the instances while binding them to the Notebook. The Name is irrelevant inside the Notebook. In the Notebook you must use %geode.oql tag.Bind to NotebookIn the Notebook click on the settings icon in the top right corner. The select/deselect the interpreters to be bound with the Notebook.ConfigurationYou can modify the configuration of the Geode from the Interpreter section.  The Geode interpreter expresses the following properties:      Property Name    Description    Default Value        geode.locator.host    The Geode Locator Host    localhost        geode.locator.port    The Geode Locator Port    10334        geode.max.result    Max number of OQL result to display to prevent the browser overload    1000  How to useTip 1: Use (CTRL + .) for OQL auto-completion.Tip 2: Always start the paragraphs with the full %geode.oql prefix tag! The short notation: %geode would still be able run the OQL queries but the syntax highlighting and the auto-completions will be disabled.Create / Destroy RegionsThe OQL specification does not support  Geode Regions mutation operations. To create/destroy regions one should use the GFSH shell tool instead. In the following it is assumed that the GFSH is colocated with Zeppelin server.%shsource /etc/geode/conf/geode-env.shgfsh << EOF connect --locator=ambari.localdomain[10334] destroy region --name=/regionEmployee destroy region --name=/regionCompany create region --name=regionEmployee --type=REPLICATE create region --name=regionCompany --type=REPLICATE exit;EOFAbove snippet re-creates two regions: regionEmployee and regionCompany. Note that you have to explicitly specify the locator host and port. The values should match those you have used in the Geode Interpreter configuration. Comprehensive list of GFSH Commands by Functional Area.Basic OQL%geode.oqlSELECT count(*) FROM /regionEmployeeOQL IN and SET filters:%geode.oqlSELECT * FROM /regionEmployeeWHERE companyId IN SET(2) OR lastName IN SET('Tzolov13', 'Tzolov73')OQL JOIN operations%geode.oqlSELECT e.employeeId, e.firstName, e.lastName, c.id as companyId, c.companyName, c.addressFROM /regionEmployee e, /regionCompany cWHERE e.companyId = c.idBy default the QOL responses contain only the region entry values. To access the keys, query the EntrySet instead:%geode.oqlSELECT e.key, e.value.companyId, e.value.emailFROM /regionEmployee.entrySet eFollowing query will return the EntrySet value as a Blob:%geode.oqlSELECT e.key, e.value FROM /regionEmployee.entrySet eNote: You can have multiple queries in the same paragraph but only the result from the first is displayed. [1], [2].GFSH Commands From The ShellUse the Shell Interpreter (%sh) to run OQL commands form the command line:%shsource /etc/geode/conf/geode-env.shgfsh -e "connect" -e "list members"Apply Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form inside your OQL queries. You can use both the text input and select form parameterization features%geode.oqlSELECT * FROM /regionEmployee e WHERE e.employeeId > ${Id}Auto-completionThe Geode Interpreter provides a basic auto-completion functionality. On (Ctrl+.) it list the most relevant suggestions in a pop-up window.Geode REST APITo list the defined regions you can use the Geode REST API:http://<geode server hostname>phd1.localdomain:8484/gemfire-api/v1/{  "regions" : [{    "name" : "regionEmployee",    "type" : "REPLICATE",    "key-constraint" : null,    "value-constraint" : null  }, {    "name" : "regionCompany",    "type" : "REPLICATE",    "key-constraint" : null,    "value-constraint" : null  }]}To enable Geode REST API with JSON support add the following properties to geode.server.properties.file and restart:http-service-port=8484start-dev-rest-api=true",
      "url": " /interpreter/geode.html",
      "group": "interpreter",
      "excerpt": "Apache Geode (incubating) provides a database-like consistency model, reliable transaction processing and a shared-nothing architecture to maintain very low latency performance with high concurrency processing."
    }
    ,
    
  

    "/interpreter/hbase.html": {
      "title": "HBase Shell Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->HBase Shell Interpreter for Apache ZeppelinOverviewHBase Shell is a JRuby IRB client for Apache HBase. This interpreter provides all capabilities of Apache HBase shell within Apache Zeppelin. The interpreter assumes that Apache HBase client software has been installed and it can connect to the Apache HBase cluster from the machine on where Apache Zeppelin is installed.To get start with HBase, please see HBase Quickstart.HBase release supportedBy default, Zeppelin is built against HBase 1.0.x releases. To work with HBase 1.1.x releases, use the following build command:# HBase 1.1.4mvn clean package -DskipTests -Phadoop-2.6 -Dhadoop.version=2.6.0 -P build-distr -Dhbase.hbase.version=1.1.4 -Dhbase.hadoop.version=2.6.0To work with HBase 1.2.0+, use the following build command:# HBase 1.2.0mvn clean package -DskipTests -Phadoop-2.6 -Dhadoop.version=2.6.0 -P build-distr -Dhbase.hbase.version=1.2.0 -Dhbase.hadoop.version=2.6.0Configuration      Property    Default    Description        hbase.home    /usr/lib/hbase    Installation directory of HBase, defaults to HBASE_HOME in environment        hbase.ruby.sources    lib/ruby    Path to Ruby scripts relative to 'hbase.home'        zeppelin.hbase.test.mode    false    Disable checks for unit and manual tests  If you want to connect to HBase running on a cluster, you'll need to follow the next step.Export HBASE_HOMEIn conf/zeppelin-env.sh, export HBASE_HOME environment variable with your HBase installation path. This ensures hbase-site.xml can be loaded.for exampleexport HBASE_HOME=/usr/lib/hbaseor, when running with CDHexport HBASE_HOME="/opt/cloudera/parcels/CDH/lib/hbase"You can optionally export HBASE_CONF_DIR instead of HBASE_HOME should you have custom HBase configurations.Enabling the HBase Shell InterpreterIn a notebook, to enable the HBase Shell interpreter, click the Gear icon and select HBase Shell.Using the HBase Shell InterpreterIn a paragraph, use %hbase to select the HBase Shell interpreter and then input all commands. To get the list of available commands, use help.%hbasehelpFor example, to create a table%hbasecreate 'test', 'cf'And then to put data into that table%hbaseput 'test', 'row1', 'cf:a', 'value1'For more information on all commands available, refer to HBase shell commands.",
      "url": " /interpreter/hbase.html",
      "group": "interpreter",
      "excerpt": "HBase Shell is a JRuby IRB client for Apache HBase. This interpreter provides all capabilities of Apache HBase shell within Apache Zeppelin."
    }
    ,
    
  

    "/interpreter/hdfs.html": {
      "title": "HDFS File System Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->HDFS File System Interpreter for Apache ZeppelinOverviewHadoop File System is a distributed, fault tolerant file system part of the hadoop project and is often used as storage for distributed processing engines like Hadoop MapReduce and Apache Spark or underlying file systems like Alluxio.Configuration      Property    Default    Description        hdfs.url    http://localhost:50070/webhdfs/v1/    The URL for WebHDFS        hdfs.user    hdfs    The WebHDFS user        hdfs.maxlength    1000    Maximum number of lines of results fetched  This interpreter connects to HDFS using the HTTP WebHDFS interface.It supports the basic shell file commands applied to HDFS, it currently only supports browsing.You can use ls [PATH] and ls -l [PATH] to list a directory. If the path is missing, then the current directory is listed.  ls  supports a -h flag for human readable file sizes.You can use cd [PATH] to change your current directory by giving a relative or an absolute path.You can invoke pwd to see your current directory.Tip : Use ( Ctrl + . ) for autocompletion.Create InterpreterIn a notebook, to enable the HDFS interpreter, click the Gear icon and select HDFS.WebHDFS REST APIYou can confirm that you're able to access the WebHDFS API by running a curl command against the WebHDFS end point provided to the interpreter.Here is an example:$> curl "http://localhost:50070/webhdfs/v1/?op=LISTSTATUS"",
      "url": " /interpreter/hdfs.html",
      "group": "interpreter",
      "excerpt": "Hadoop File System is a distributed, fault tolerant file system part of the hadoop project and is often used as storage for distributed processing engines like Hadoop MapReduce and Apache Spark or underlying file systems like Alluxio."
    }
    ,
    
  

    "/interpreter/hive.html": {
      "title": "Hive Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Hive Interpreter for Apache ZeppelinImportant NoticeHive Interpreter will be deprecated and merged into JDBC Interpreter. You can use Hive Interpreter by using JDBC Interpreter with same functionality. See the example below of settings and dependencies.Properties      Property    Value        hive.driver    org.apache.hive.jdbc.HiveDriver        hive.url    jdbc:hive2://localhost:10000        hive.user    hiveUser        hive.password    hivePassword  Dependencies      Artifact    Exclude        org.apache.hive:hive-jdbc:0.14.0            org.apache.hadoop:hadoop-common:2.6.0      Configuration      Property    Default    Description        default.driver    org.apache.hive.jdbc.HiveDriver    Class path of JDBC driver        default.url    jdbc:hive2://localhost:10000    Url for connection        default.user        ( Optional ) Username of the connection        default.password        ( Optional ) Password of the connection        default.xxx        ( Optional ) Other properties used by the driver        ${prefix}.driver        Driver class path of %hive(${prefix})         ${prefix}.url        Url of %hive(${prefix})         ${prefix}.user        ( Optional ) Username of the connection of %hive(${prefix})         ${prefix}.password        ( Optional ) Password of the connection of %hive(${prefix})         ${prefix}.xxx        ( Optional ) Other properties used by the driver of %hive(${prefix})   This interpreter provides multiple configuration with ${prefix}. User can set a multiple connection properties by this prefix. It can be used like %hive(${prefix}).OverviewThe Apache Hive ™ data warehouse software facilitates querying and managing large datasets residing in distributed storage. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this language also allows traditional map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefficient to express this logic in HiveQL.How to useBasically, you can use%hiveselect * from my_table;or%hive(etl)-- 'etl' is a ${prefix}select * from my_table;You can also run multiple queries up to 10 by default. Changing these settings is not implemented yet.Apply Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form inside your queries. You can use both the text input and select form parameterization features.%hiveSELECT ${group_by}, count(*) as countFROM retail_demo.order_lineitems_pxfGROUP BY ${group_by=product_id,product_id|product_name|customer_id|store_id}ORDER BY count ${order=DESC,DESC|ASC}LIMIT ${limit=10};",
      "url": " /interpreter/hive.html",
      "group": "interpreter",
      "excerpt": "Apache Hive data warehouse software facilitates querying and managing large datasets residing in distributed storage. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this..."
    }
    ,
    
  

    "/interpreter/ignite.html": {
      "title": "Ignite Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Ignite Interpreter for Apache ZeppelinOverviewApache Ignite In-Memory Data Fabric is a high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash technologies.You can use Zeppelin to retrieve distributed data from cache using Ignite SQL interpreter. Moreover, Ignite interpreter allows you to execute any Scala code in cases when SQL doesn't fit to your requirements. For example, you can populate data into your caches or execute distributed computations.Installing and Running Ignite exampleIn order to use Ignite interpreters, you may install Apache Ignite in some simple steps:Ignite provides examples only with source or binary release. Download Ignite source release or binary release whatever you want. But you must download Ignite as the same version of Zeppelin's. If it is not, you can't use scala code on Zeppelin. The supported Ignite version is specified in Supported Interpreter table for each Zeppelin release. If you're using Zeppelin master branch, please see ignite.version in path/to/your-Zeppelin/ignite/pom.xml.Examples are shipped as a separate Maven project, so to start running you simply need to import provided <dest_dir>/apache-ignite-fabric-{version}-bin/examples/pom.xml file into your favourite IDE, such as Eclipse.In case of Eclipse, Eclipse -> File -> Import -> Existing Maven ProjectsSet examples directory path to Eclipse and select the pom.xml.Then start org.apache.ignite.examples.ExampleNodeStartup (or whatever you want) to run at least one or more ignite node. When you run example code, you may notice that the number of node is increase one by one.Tip. If you want to run Ignite examples on the cli not IDE, you can export executable Jar file from IDE. Then run it by using below command.$ nohup java -jar </path/to/your Jar file name>Configuring Ignite InterpreterAt the "Interpreters" menu, you may edit Ignite interpreter or create new one. Zeppelin provides these properties for Ignite.      Property Name    value    Description        ignite.addresses    127.0.0.1:47500..47509    Coma separated list of Ignite cluster hosts. See [Ignite Cluster Configuration](https://apacheignite.readme.io/docs/cluster-config) section for more details.        ignite.clientMode    true    You can connect to the Ignite cluster as client or server node. See [Ignite Clients vs. Servers](https://apacheignite.readme.io/docs/clients-vs-servers) section for details. Use true or false values in order to connect in client or server mode respectively.        ignite.config.url        Configuration URL. Overrides all other settings.        ignite.jdbc.url    jdbc:ignite:cfg://default-ignite-jdbc.xml    Ignite JDBC connection URL.        ignite.peerClassLoadingEnabled    true    Enables peer-class-loading. See [Zero Deployment](https://apacheignite.readme.io/docs/zero-deployment) section for details. Use true or false values in order to enable or disable P2P class loading respectively.  How to useAfter configuring Ignite interpreter, create your own notebook. Then you can bind interpreters like below image.For more interpreter binding information see here.Ignite SQL interpreterIn order to execute SQL query, use %ignite.ignitesql prefix. Supposing you are running org.apache.ignite.examples.streaming.wordcount.StreamWords, then you can use "words" cache( Of course you have to specify this cache name to the Ignite interpreter setting section ignite.jdbc.url of Zeppelin ).For example, you can select top 10 words in the words cache using the following query%ignite.ignitesqlselect _val, count(_val) as cnt from String group by _val order by cnt desc limit 10As long as your Ignite version and Zeppelin Ignite version is same, you can also use scala code. Please check the Zeppelin Ignite version before you download your own Ignite.%igniteimport org.apache.ignite._import org.apache.ignite.cache.affinity._import org.apache.ignite.cache.query._import org.apache.ignite.configuration._import scala.collection.JavaConversions._val cache: IgniteCache[AffinityUuid, String] = ignite.cache("words")val qry = new SqlFieldsQuery("select avg(cnt), min(cnt), max(cnt) from (select count(_val) as cnt from String group by _val)", true)val res = cache.query(qry).getAll()collectionAsScalaIterable(res).foreach(println _)Apache Ignite also provides a guide docs for Zeppelin "Ignite with Apache Zeppelin"",
      "url": " /interpreter/ignite.html",
      "group": "interpreter",
      "excerpt": "Apache Ignite in-memory Data Fabric is a high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash technologies."
    }
    ,
    
  

    "/interpreter/jdbc.html": {
      "title": "Generic JDBC Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Generic JDBC Interpreter for Apache ZeppelinOverviewJDBC interpreter lets you create a JDBC connection to any data sources seamlessly.Inserts, Updates, and Upserts are applied immediately after running each statement.By now, it has been tested with:                    Postgresql -      JDBC Driver              Mysql -      JDBC Driver              MariaDB -      JDBC Driver              Redshift -      JDBC Driver              Apache Hive -       JDBC Driver              Apache Phoenix itself is a JDBC driver              Apache Drill -       JDBC Driver              Apache Tajo -       JDBC Driver      If you are using other databases not in the above list, please feel free to share your use case. It would be helpful to improve the functionality of JDBC interpreter.Create a new JDBC InterpreterFirst, click + Create button at the top-right corner in the interpreter setting page.Fill Interpreter name field with whatever you want to use as the alias(e.g. mysql, mysql2, hive, redshift, and etc..). Please note that this alias will be used as %interpreter_name to call the interpreter in the paragraph. Then select jdbc as an Interpreter group. The default driver of JDBC interpreter is set as PostgreSQL. It means Zeppelin includes PostgreSQL driver jar in itself.So you don't need to add any dependencies(e.g. the artifact name or path for PostgreSQL driver jar) for PostgreSQL connection.The JDBC interpreter properties are defined by default like below.      Name    Default Value    Description        common.max_count    1000    The maximun number of SQL result to display        default.driver    org.postgresql.Driver    JDBC Driver Name        default.password        The JDBC user password        default.url    jdbc:postgresql://localhost:5432/    The URL for JDBC        default.user    gpadmin    The JDBC user name  If you want to connect other databases such as Mysql, Redshift and Hive, you need to edit the property values.You can also use Credential for JDBC authentication.If default.user and default.password properties are deleted(using X button) for database connection in the interpreter setting page,the JDBC interpreter will get the account information from Credential.The below example is for Mysql connection.The last step is Dependency Setting. Since Zeppelin only includes PostgreSQL driver jar by default, you need to add each driver's maven coordinates or JDBC driver's jar file path for the other databases.That's it. You can find more JDBC connection setting examples(Mysql, MariaDB, Redshift, Apache Hive, Apache Phoenix, and Apache Tajo) in this section.More propertiesThere are more JDBC interpreter properties you can specify like below.      Property Name    Description        common.max_result    Max number of SQL result to display to prevent the browser overload. This is  common properties for all connections        zeppelin.jdbc.auth.type    Types of authentications' methods supported are SIMPLE, and KERBEROS        zeppelin.jdbc.principal    The principal name to load from the keytab        zeppelin.jdbc.keytab.location    The path to the keytab file          zeppelin.jdbc.auth.kerberos.proxy.enable      When auth type is Kerberos, enable/disable Kerberos proxy with the login user to get the connection. Default value is true.        default.jceks.file    jceks store path (e.g: jceks://file/tmp/zeppelin.jceks)        default.jceks.credentialKey    jceks credential key  You can also add more properties by using this method.For example, if a connection needs a schema parameter, it would have to add the property as follows:      name    value        default.schema    schema_name  Binding JDBC interpter to notebookTo bind the interpreters created in the interpreter setting page, click the gear icon at the top-right corner.Select(blue) or deselect(white) the interpreter buttons depending on your use cases. If you need to use more than one interpreter in the notebook, activate several buttons.Don't forget to click Save button, or you will face Interpreter *** is not found error.How to useRun the paragraph with JDBC interpreterTo test whether your databases and Zeppelin are successfully connected or not, type %jdbc_interpreter_name(e.g. %mysql) at the top of the paragraph and run show databases.%jdbc_interpreter_nameshow databasesIf the paragraph is FINISHED without any errors, a new paragraph will be automatically added after the previous one with %jdbc_interpreter_name.So you don't need to type this prefix in every paragraphs' header.Apply Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form inside your queries. You can use both the text input and select form parametrization features.%jdbc_interpreter_nameSELECT name, country, performerFROM demo.performersWHERE name='{{performer=Sheryl Crow|Doof|Fanfarlo|Los Paranoia}}'ExamplesHere are some examples you can refer to. Including the below connectors, you can connect every databases as long as it can be configured with it's JDBC driver.PostgresProperties      Name    Value        default.driver    org.postgresql.Driver        default.url    jdbc:postgresql://localhost:5432/        default.user    mysql_user        default.password    mysql_password  Postgres JDBC Driver DocsDependencies      Artifact    Excludes        org.postgresql:postgresql:9.4.1211      Maven Repository: org.postgresql:postgresqlMysqlProperties      Name    Value        default.driver    com.mysql.jdbc.Driver        default.url    jdbc:mysql://localhost:3306/        default.user    mysql_user        default.password    mysql_password  Mysql JDBC Driver DocsDependencies      Artifact    Excludes        mysql:mysql-connector-java:5.1.38      Maven Repository: mysql:mysql-connector-javaMariaDBProperties      Name    Value        default.driver    org.mariadb.jdbc.Driver        default.url    jdbc:mariadb://localhost:3306        default.user    mariadb_user        default.password    mariadb_password  MariaDB JDBC Driver DocsDependencies      Artifact    Excludes        org.mariadb.jdbc:mariadb-java-client:1.5.4      Maven Repository: org.mariadb.jdbc:mariadb-java-clientRedshiftProperties      Name    Value        default.driver    com.amazon.redshift.jdbc42.Driver        default.url    jdbc:redshift://your-redshift-instance-address.redshift.amazonaws.com:5439/your-database        default.user    redshift_user        default.password    redshift_password  AWS Redshift JDBC Driver DocsDependencies      Artifact    Excludes        com.amazonaws:aws-java-sdk-redshift:1.11.51      Maven Repository: com.amazonaws:aws-java-sdk-redshiftApache HiveProperties      Name    Value        default.driver    org.apache.hive.jdbc.HiveDriver        default.url    jdbc:hive2://localhost:10000        default.user    hive_user        default.password    hive_password  Apache Hive 1 JDBC Driver DocsApache Hive 2 JDBC Driver DocsDependencies      Artifact    Excludes        org.apache.hive:hive-jdbc:0.14.0            org.apache.hadoop:hadoop-common:2.6.0      Maven Repository : org.apache.hive:hive-jdbcApache PhoenixPhoenix supports thick and thin connection types:Thick client is faster, but must connect directly to ZooKeeper and HBase RegionServers.Thin client has fewer dependencies and connects through a Phoenix Query Server instance.Use the appropriate default.driver, default.url, and the dependency artifact for your connection type.Thick client connectionProperties      Name    Value        default.driver    org.apache.phoenix.jdbc.PhoenixDriver        default.url    jdbc:phoenix:localhost:2181:/hbase-unsecure        default.user    phoenix_user        default.password    phoenix_password  Dependencies      Artifact    Excludes        org.apache.phoenix:phoenix-core:4.4.0-HBase-1.0      Maven Repository: org.apache.phoenix:phoenix-coreThin client connectionProperties      Name    Value        default.driver    org.apache.phoenix.queryserver.client.Driver        default.url    jdbc:phoenix:thin:url=http://localhost:8765;serialization=PROTOBUF        default.user    phoenix_user        default.password    phoenix_password  DependenciesBefore Adding one of the below dependencies, check the Phoenix version first.      Artifact    Excludes    Description        org.apache.phoenix:phoenix-server-client:4.7.0-HBase-1.1        For Phoenix 4.7        org.apache.phoenix:phoenix-queryserver-client:4.8.0-HBase-1.2        For Phoenix 4.8+  Maven Repository: org.apache.phoenix:phoenix-queryserver-clientApache TajoProperties      Name    Value        default.driver    org.apache.tajo.jdbc.TajoDriver        default.url    jdbc:tajo://localhost:26002/default  Apache Tajo JDBC Driver DocsDependencies      Artifact    Excludes        org.apache.tajo:tajo-jdbc:0.11.0      Maven Repository: org.apache.tajo:tajo-jdbcBug reportingIf you find a bug using JDBC interpreter, please create a JIRA ticket.",
      "url": " /interpreter/jdbc.html",
      "group": "interpreter",
      "excerpt": "Generic JDBC Interpreter lets you create a JDBC connection to any data source. You can use Postgres, MySql, MariaDB, Redshift, Apache Hive, Apache Phoenix, Apache Drill and Apache Tajo using JDBC interpreter."
    }
    ,
    
  

    "/interpreter/kylin.html": {
      "title": "Apache Kylin Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Kylin Interpreter for Apache ZeppelinOverviewApache Kylin is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets, original contributed from eBay Inc. The interpreter assumes that Apache Kylin has been installed and you can connect to Apache Kylin from the machine Apache Zeppelin is installed.To get start with Apache Kylin, please see Apache Kylin Quickstart.Configuration      Name    Default    Description        kylin.api.url     http://localhost:7070/kylin/api/query    kylin query POST API  The format can be like http://<host>:<port>/kylin/api/query        kylin.api.user    ADMIN    kylin user        kylin.api.password    KYLIN    kylin password        kylin.query.project    learn_kylin    String, Project to perform query. Could update at notebook level        kylin.query.ispartial    true    true|false  (@Deprecated since Apache Kylin V1.5)  Whether accept a partial result or not, default be “false”. Set to “false” for production use.        kylin.query.limit    5000    int, Query limit  If limit is set in sql, perPage will be ignored.        kylin.query.offset    0    int, Query offset  If offset is set in sql, curIndex will be ignored.  Using the Apache Kylin InterpreterIn a paragraph, use %kylin(project_name) to select the kylin interpreter, project name and then input sql. If no project name defined, will use the default project name from the above configuration.%kylin(learn_project)select count(*) from kylin_sales group by part_dt",
      "url": " /interpreter/kylin.html",
      "group": "interpreter",
      "excerpt": "Apache Kylin™ is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets, original contributed from eBay Inc. ."
    }
    ,
    
  

    "/interpreter/lens.html": {
      "title": "Lens Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Lens Interpreter for Apache ZeppelinOverviewApache Lens provides an Unified Analytics interface. Lens aims to cut the Data Analytics silos by providing a single view of data across multiple tiered data stores and optimal execution environment for the analytical query. It seamlessly integrates Hadoop with traditional data warehouses to appear like one.Installing and Running LensIn order to use Lens interpreters, you may install Apache Lens in some simple steps:Download Lens for latest version from the ASF. Or the older release can be found in the Archives.Before running Lens, you have to set HIVEHOME and HADOOPHOME. If you want to get more information about this, please refer to here. Lens also provides Pseudo Distributed mode. Lens pseudo-distributed setup is done by using docker. Hive server and hadoop daemons are run as separate processes in lens pseudo-distributed setup.Now, you can start lens server (or stop)../bin/lens-ctl start (or stop)Configuring Lens InterpreterAt the "Interpreters" menu, you can edit Lens interpreter or create new one. Zeppelin provides these properties for Lens.      Property Name    value    Description        lens.client.dbname    default    The database schema name        lens.query.enable.persistent.resultset    false    Whether to enable persistent resultset for queries. When enabled, server will fetch results from driver, custom format them if any and store in a configured location. The file name of query output is queryhandle-id, with configured extensions        lens.server.base.url    http://hostname:port/lensapi    The base url for the lens server. you have to edit "hostname" and "port" that you may use(ex. http://0.0.0.0:9999/lensapi)          lens.session.cluster.user     default    Hadoop cluster username        zeppelin.lens.maxResult    1000    Max number of rows to display        zeppelin.lens.maxThreads    10    If concurrency is true then how many threads?        zeppelin.lens.run.concurrent    true    Run concurrent Lens Sessions        xxx    yyy    anything else from [Configuring lens server](https://lens.apache.org/admin/config-server.html)  Interpreter Binding for Zeppelin NotebookAfter configuring Lens interpreter, create your own notebook, then you can bind interpreters like below image.For more interpreter binding information see here.How to useYou can analyze your data by using OLAP Cube QL which is a high level SQL like language to query and describe data sets organized in data cubes.You may experience OLAP Cube like this Video tutorial.As you can see in this video, they are using Lens Client Shell(./bin/lens-cli.sh). All of these functions also can be used on Zeppelin by using Lens interpreter. Create and Use(Switch) Databases.create database newDbuse newDb Create Storage.create storage your/path/to/lens/client/examples/resources/db-storage.xml Create Dimensions, Show fields and join-chains of them.create dimension your/path/to/lens/client/examples/resources/customer.xmldimension show fields customerdimension show joinchains customer Create Caches, Show fields and join-chains of them.create cube your/path/to/lens/client/examples/resources/sales-cube.xmlcube show fields salescube show joinchains sales Create Dimtables and Fact.create dimtable your/path/to/lens/client/examples/resources/customer_table.xmlcreate fact your/path/to/lens/client/examples/resources/sales-raw-fact.xml Add partitions to Dimtable and Fact.dimtable add single-partition --dimtable_name customer_table --storage_name local --path your/path/to/lens/client/examples/resources/customer-local-part.xmlfact add partitions --fact_name sales_raw_fact --storage_name local --path your/path/to/lens/client/examples/resources/sales-raw-local-parts.xml Now, you can run queries on cubes.query execute cube select customer_city_name, product_details.description, product_details.category, product_details.color, store_sales from sales where time_range_in(delivery_time, '2015-04-11-00', '2015-04-13-00')These are just examples that provided in advance by Lens. If you want to explore whole tutorials of Lens, see the tutorial video.Lens UI ServiceLens also provides web UI service. Once the server starts up, you can open the service on http://serverhost:19999/index.html and browse. You may also check the structure that you made and use query easily here.",
      "url": " /interpreter/lens.html",
      "group": "interpreter",
      "excerpt": "Apache Lens provides an Unified Analytics interface. Lens aims to cut the Data Analytics silos by providing a single view of data across multiple tiered data stores and optimal execution environment for the analytical query. It seamlessly integrates Hadoop with..."
    }
    ,
    
  

    "/interpreter/livy.html": {
      "title": "Livy Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Livy Interpreter for Apache ZeppelinOverviewLivy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN.Interactive Scala, Python and R shellsBatch submissions in Scala, Java, PythonMulti users can share the same server (impersonation support)Can be used for submitting jobs from anywhere with RESTDoes not require any code change to your programsRequirementsAdditional requirements for the Livy interpreter are:Spark 1.3 or above.Livy server.ConfigurationWe added some common configurations for spark, and you can set any configuration you want.You can find all Spark configurations in here.And instead of starting property with spark. it should be replaced with livy.spark..Example: spark.driver.memory to livy.spark.driver.memory      Property    Default    Description        zeppelin.livy.url    http://localhost:8998    URL where livy server is running        zeppelin.livy.spark.maxResult    1000    Max number of Spark SQL result to display.        zeppelin.livy.session.create_timeout    120    Timeout in seconds for session creation        zeppelin.livy.displayAppInfo    false    Whether to display app info        zeppelin.livy.pull_status.interval.millis    1000    The interval for checking paragraph execution status        livy.spark.driver.cores        Driver cores. ex) 1, 2.          livy.spark.driver.memory        Driver memory. ex) 512m, 32g.          livy.spark.executor.instances        Executor instances. ex) 1, 4.          livy.spark.executor.cores        Num cores per executor. ex) 1, 4.        livy.spark.executor.memory        Executor memory per worker instance. ex) 512m, 32g.        livy.spark.dynamicAllocation.enabled        Use dynamic resource allocation. ex) True, False.        livy.spark.dynamicAllocation.cachedExecutorIdleTimeout        Remove an executor which has cached data blocks.        livy.spark.dynamicAllocation.minExecutors        Lower bound for the number of executors.        livy.spark.dynamicAllocation.initialExecutors        Initial number of executors to run.        livy.spark.dynamicAllocation.maxExecutors        Upper bound for the number of executors.            livy.spark.jars.packages            Adding extra libraries to livy interpreter          zeppelin.livy.ssl.trustStore        client trustStore file. Used when livy ssl is enabled        zeppelin.livy.ssl.trustStorePassword        password for trustStore file. Used when livy ssl is enabled    We remove livy.spark.master in zeppelin-0.7. Because we sugguest user to use livy 0.3 in zeppelin-0.7. And livy 0.3 don't allow to specify livy.spark.master, it enfornce yarn-cluster mode.Adding External librariesYou can load dynamic library to livy interpreter by set livy.spark.jars.packages property to comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. The format for the coordinates should be groupId:artifactId:version. Example      Property    Example    Description          livy.spark.jars.packages      io.spray:spray-json_2.10:1.3.1      Adding extra libraries to livy interpreter        How to useBasically, you can usespark%livy.sparksc.versionpyspark%livy.pysparkprint "1"sparkR%livy.sparkrhello <- function( name ) {    sprintf( "Hello, %s", name );}hello("livy")ImpersonationWhen Zeppelin server is running with authentication enabled, then this interpreter utilizes Livy’s user impersonation feature i.e. sends extra parameter for creating and running a session ("proxyUser": "${loggedInUser}"). This is particularly useful when multi users are sharing a Notebook server.Apply Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form. You can use both the text input and select form parameterization features.%livy.pysparkprint "${group_by=product_id,product_id|product_name|customer_id|store_id}"FAQLivy debugging: If you see any of these in error consoleConnect to livyhost:8998 [livyhost/127.0.0.1, livyhost/0:0:0:0:0:0:0:1] failed: Connection refusedLooks like the livy server is not up yet or the config is wrongException: Session not found, Livy server would have restarted, or lost session.The session would have timed out, you may need to restart the interpreter.Blacklisted configuration values in session config: spark.masterEdit conf/spark-blacklist.conf file in livy server and comment out #spark.master line.If you choose to work on livy in apps/spark/java directory in https://github.com/cloudera/hue,copy spark-user-configurable-options.template to spark-user-configurable-options.conf file in livy server and comment out #spark.master. ",
      "url": " /interpreter/livy.html",
      "group": "interpreter",
      "excerpt": "Livy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN."
    }
    ,
    
  

    "/interpreter/mahout.html": {
      "title": "Mahout Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Mahout Interpreter for Apache ZeppelinInstallationApache Mahout is a collection of packages that enable machine learning and matrix algebra on underlying engines such as Apache Flink or Apache Spark.  A convenience script for creating and configuring two Mahout enabled interpreters exists.  The %sparkMahout and %flinkMahout interpreters do not exist by default but can be easily created using this script.  Easy InstallationTo quickly and easily get up and running using Apache Mahout, run the following command from the top-level directory of the Zeppelin install:bashpython scripts/mahout/add_mahout.pyThis will create the %sparkMahout and %flinkMahout interpreters, and restart Zeppelin.Advanced InstallationThe add_mahout.py script contains several command line arguments for advanced users.      Argument    Description    Example        --zeppelin_home    This is the path to the Zeppelin installation.  This flag is not needed if the script is run from the top-level installation directory or from the `zeppelin/scripts/mahout` directory.    /path/to/zeppelin        --mahout_home    If the user has already installed Mahout, this flag can set the path to `MAHOUT_HOME`.  If this is set, downloading Mahout will be skipped.    /path/to/mahout_home        --restart_later    Restarting is necessary for updates to take effect. By default the script will restart Zeppelin for you- restart will be skipped if this flag is set.    NA        --force_download    This flag will force the script to re-download the binary even if it already exists.  This is useful for previously failed downloads.    NA          --overwrite_existing      This flag will force the script to overwrite existing `%sparkMahout` and `%flinkMahout` interpreters. Useful when you want to just start over.      NA    NOTE 1: Apache Mahout at this time only supports Spark 1.5 and Spark 1.6 and Scala 2.10.  If the user is using another version of Spark (e.g. 2.0), the %sparkMahout will likely not work.  The %flinkMahout interpreter will still work and the user is encouraged to develop with that engine as the code can be ported via copy and paste, as is evidenced by the tutorial notebook.NOTE 2: If using Apache Flink in cluster mode, the following libraries will also need to be coppied to ${FLINK_HOME}/lib- mahout-math-0.12.2.jar- mahout-math-scala2.10-0.12.2.jar- mahout-flink2.10-0.12.2.jar- mahout-hdfs-0.12.2.jar- com.google.guava:guava:14.0.1OverviewThe Apache Mahout™ project's goal is to build an environment for quickly creating scalable performant machine learning applications.Apache Mahout software provides three major features:A simple and extensible programming environment and framework for building scalable algorithmsA wide variety of premade algorithms for Scala + Apache Spark, H2O, Apache FlinkSamsara, a vector math experimentation environment with R-like syntax which works at scaleIn other words:Apache Mahout provides a unified API for quickly creating machine learning algorithms on a variety of engines.How to useWhen starting a session with Apache Mahout, depending on which engine you are using (Spark or Flink), a few imports must be made and a Distributed Context must be declared.  Copy and paste the following code and run once to get started.Flink%flinkMahoutimport org.apache.flink.api.scala._import org.apache.mahout.math.drm._import org.apache.mahout.math.drm.RLikeDrmOps._import org.apache.mahout.flinkbindings._import org.apache.mahout.math._import scalabindings._import RLikeOps._implicit val ctx = new FlinkDistributedContext(benv)Spark%sparkMahoutimport org.apache.mahout.math._import org.apache.mahout.math.scalabindings._import org.apache.mahout.math.drm._import org.apache.mahout.math.scalabindings.RLikeOps._import org.apache.mahout.math.drm.RLikeDrmOps._import org.apache.mahout.sparkbindings._implicit val sdc: org.apache.mahout.sparkbindings.SparkDistributedContext = sc2sdc(sc)Same Code, Different EnginesAfter importing and setting up the distributed context, the Mahout R-Like DSL is consistent across engines.  The following code will run in both %flinkMahout and %sparkMahoutval drmData = drmParallelize(dense(  (2, 2, 10.5, 10, 29.509541),  // Apple Cinnamon Cheerios  (1, 2, 12,   12, 18.042851),  // Cap'n'Crunch  (1, 1, 12,   13, 22.736446),  // Cocoa Puffs  (2, 1, 11,   13, 32.207582),  // Froot Loops  (1, 2, 12,   11, 21.871292),  // Honey Graham Ohs  (2, 1, 16,   8,  36.187559),  // Wheaties Honey Gold  (6, 2, 17,   1,  50.764999),  // Cheerios  (3, 2, 13,   7,  40.400208),  // Clusters  (3, 3, 13,   4,  45.811716)), numPartitions = 2)drmData.collect(::, 0 until 4)val drmX = drmData(::, 0 until 4)val y = drmData.collect(::, 4)val drmXtX = drmX.t %*% drmXval drmXty = drmX.t %*% yval XtX = drmXtX.collectval Xty = drmXty.collect(::, 0)val beta = solve(XtX, Xty)Leveraging Resource Pools and R for VisualizationResource Pools are a powerful Zeppelin feature that lets us share information between interpreters. A fun trick is to take the output of our work in Mahout and analyze it in other languages.Setting up a Resource Pool in FlinkIn Spark based interpreters resource pools are accessed via the ZeppelinContext API.  To put and get things from the resource pool one can be done simplescalaval myVal = 1z.put("foo", myVal)val myFetchedVal = z.get("foo")To add this functionality to a Flink based interpreter we declare the follwoing%flinkMahoutimport org.apache.zeppelin.interpreter.InterpreterContextval z = InterpreterContext.get().getResourcePool()Now we can access the resource pool in a consistent manner from the %flinkMahout interpreter.Passing a variable from Mahout to R and PlottingIn this simple example, we use Mahout (on Flink or Spark, the code is the same) to create a random matrix and then take the Sin of each element. We then randomly sample the matrix and create a tab separated string. Finally we pass that string to R where it is read as a .tsv file, and a DataFrame is created and plotted using native R plotting libraries.val mxRnd = Matrices.symmetricUniformView(5000, 2, 1234)val drmRand = drmParallelize(mxRnd)val drmSin = drmRand.mapBlock() {case (keys, block) =>    val blockB = block.like()  for (i <- 0 until block.nrow) {    blockB(i, 0) = block(i, 0)    blockB(i, 1) = Math.sin((block(i, 0) * 8))  }  keys -> blockB}z.put("sinDrm", org.apache.mahout.math.drm.drmSampleToTSV(drmSin, 0.85))And then in an R paragraph...%spark.r {"imageWidth": "400px"}library("ggplot2")sinStr = z.get("flinkSinDrm")data <- read.table(text= sinStr, sep="t", header=FALSE)plot(data,  col="red")",
      "url": " /interpreter/mahout.html",
      "group": "interpreter",
      "excerpt": "Apache Mahout provides a unified API (the R-Like Scala DSL) for quickly creating machine learning algorithms on a variety of engines."
    }
    ,
    
  

    "/interpreter/markdown.html": {
      "title": "Markdown Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Markdown Interpreter for Apache ZeppelinOverviewMarkdown is a plain text formatting syntax designed so that it can be converted to HTML.Apache Zeppelin uses pegdown and markdown4j as markdown parsers.In Zeppelin notebook, you can use %md in the beginning of a paragraph to invoke the Markdown interpreter and generate static html from Markdown plain text.In Zeppelin, Markdown interpreter is enabled by default and uses the pegdown parser.ExampleThe following example demonstrates the basic usage of Markdown in a Zeppelin notebook.Mathematical expressionMarkdown interpreter leverages %html display system internally. That means you can mix mathematical expressions with markdown syntax. For more information, please see Mathematical Expression section.Configuration      Name    Default Value    Description        markdown.parser.type    pegdown    Markdown Parser Type.  Available values: pegdown, markdown4j.  Pegdown Parserpegdown parser provides github flavored markdown.pegdown parser provides YUML and Websequence plugins also. Markdown4j ParserSince pegdown parser is more accurate and provides much more markdown syntaxmarkdown4j option might be removed later. But keep this parser for the backward compatibility.",
      "url": " /interpreter/markdown.html",
      "group": "interpreter",
      "excerpt": "Markdown is a plain text formatting syntax designed so that it can be converted to HTML. Apache Zeppelin uses markdown4j."
    }
    ,
    
  

    "/interpreter/pig.html": {
      "title": "Pig Interpreter for Apache Zeppelin",
      "content"  : "Pig Interpreter for Apache ZeppelinOverviewApache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets.Supported interpreter type%pig.script (default Pig interpreter, so you can use %pig)%pig.script is like the Pig grunt shell. Anything you can run in Pig grunt shell can be run in %pig.script interpreter, it is used for running Pig script where you don’t need to visualize the data, it is suitable for data munging. %pig.query%pig.query is a little different compared with %pig.script. It is used for exploratory data analysis via Pig latin where you can leverage Zeppelin’s visualization ability. There're 2 minor differences in the last statement between %pig.script and %pig.queryNo pig alias in the last statement in %pig.query (read the examples below).The last statement must be in single line in %pig.querySupported runtime modeLocalMapReduceTez_Local (Only Tez 0.7 is supported)Tez  (Only Tez 0.7 is supported)How to useHow to setup PigLocal ModeNothing needs to be done for local modeMapReduce ModeHADOOP_CONF_DIR needs to be specified in ZEPPELIN_HOME/conf/zeppelin-env.sh.Tez Local ModeNothing needs to be done for tez local modeTez ModeHADOOP_CONF_DIR and TEZ_CONF_DIR needs to be specified in ZEPPELIN_HOME/conf/zeppelin-env.sh.How to configure interpreterAt the Interpreters menu, you have to create a new Pig interpreter. Pig interpreter has below properties by default.And you can set any Pig properties here which will be passed to Pig engine. (like tez.queue.name & mapred.job.queue.name).Besides, we use paragraph title as job name if it exists, else use the last line of Pig script. So you can use that to find app running in YARN RM UI.            Property        Default        Description                zeppelin.pig.execType        mapreduce        Execution mode for pig runtime. local | mapreduce | tez_local | tez                 zeppelin.pig.includeJobStats        false        whether display jobStats info in %pig.script                zeppelin.pig.maxResult        1000        max row number displayed in %pig.query                tez.queue.name        default        queue name for tez engine                mapred.job.queue.name        default        queue name for mapreduce engine      Examplepig%pigbankText = load 'bank.csv' using PigStorage(';');bank = foreach bankText generate $0 as age, $1 as job, $2 as marital, $3 as education, $5 as balance; bank = filter bank by age != '"age"';bank = foreach bank generate (int)age, REPLACE(job,'"','') as job, REPLACE(marital, '"', '') as marital, (int)(REPLACE(balance, '"', '')) as balance;store bank into 'clean_bank.csv' using PigStorage(';'); -- this statement is optional, it just show you that most of time %pig.script is used for data munging before querying the data. pig.queryGet the number of each age where age is less than 30%pig.querybank_data = filter bank by age < 30;b = group bank_data by age;foreach b generate group, COUNT($1);The same as above, but use dynamic text form so that use can specify the variable maxAge in textbox. (See screenshot below). Dynamic form is a very cool feature of Zeppelin, you can refer this link) for details.%pig.querybank_data = filter bank by age < ${maxAge=40};b = group bank_data by age;foreach b generate group, COUNT($1) as count;Get the number of each age for specific marital type, also use dynamic form here. User can choose the marital type in the dropdown list (see screenshot below).%pig.querybank_data = filter bank by marital=='${marital=single,single|divorced|married}';b = group bank_data by age;foreach b generate group, COUNT($1) as count;The above examples are in the Pig tutorial note in Zeppelin, you can check that for details. Here's the screenshot.Data is shared between %pig and %pig.query, so that you can do some common work in %pig, and do different kinds of query based on the data of %pig. Besides, we recommend you to specify alias explicitly so that the visualization can display the column name correctly. In the above example 2 and 3 of %pig.query, we name COUNT($1) as count. If you don't do this,then we will name it using position. E.g. in the above first example of %pig.query, we will use col_1 in chart to represent COUNT($1).",
      "url": " /interpreter/pig.html",
      "group": "manual",
      "excerpt": "Apache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs."
    }
    ,
    
  

    "/interpreter/postgresql.html": {
      "title": "PostgreSQL, Apache HAWQ (incubating) Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->PostgreSQL, Apache HAWQ (incubating) Interpreter for Apache ZeppelinImportant NoticePostgresql Interpreter will be deprecated and merged into JDBC Interpreter. You can use Postgresql by using JDBC Interpreter with same functionality. See the example below of settings and dependencies.Properties      Property    Value        psql.driver    org.postgresql.Driver        psql.url    jdbc:postgresql://localhost:5432/        psql.user    psqlUser        psql.password    psqlPassword  Dependencies      Artifact    Exclude        org.postgresql:postgresql:9.4-1201-jdbc41      OverviewThis interpreter seamlessly supports the following SQL data processing engines:PostgreSQL - OSS, Object-relational database management system (ORDBMS)pache HAWQ (incubating) - Powerful open source SQL-On-Hadoop engine.Greenplum - MPP database built on open source PostgreSQL.This Video Tutorial illustrates some of the features provided by the Postgresql Interpreter.      Name    Class    Description        %psql.sql    PostgreSqlInterpreter    Provides SQL environment for PostgreSQL, HAWQ and Greenplum  Create InterpreterBy default Zeppelin creates one PSQL instance. You can remove it or create new instances.Multiple PSQL instances can be created, each configured to the same or different backend databases. But over time a  Notebook can have only one PSQL interpreter instance bound. That means you cannot connect to different databases in the same Notebook. This is a known Zeppelin limitation.To create new PSQL instance open the Interpreter section and click the +Create button. Pick a Name of your choice and from the Interpreter drop-down select psql.  Then follow the configuration instructions and Save the new instance.Note: The Name of the instance is used only to distinct the instances while binding them to the Notebook. The Name is irrelevant inside the Notebook. In the Notebook you must use %psql.sql tag.Bind to NotebookIn the Notebook click on the settings icon in the top right corner. The select/deselect the interpreters to be bound with the Notebook.ConfigurationYou can modify the configuration of the PSQL from the Interpreter section.  The PSQL interpreter expenses the following properties:      Property Name    Description    Default Value        postgresql.url    JDBC URL to connect to     jdbc:postgresql://localhost:5432        postgresql.user    JDBC user name    gpadmin        postgresql.password    JDBC password            postgresql.driver.name    JDBC driver name. In this version the driver name is fixed and should not be changed    org.postgresql.Driver        postgresql.max.result    Max number of SQL result to display to prevent the browser overload    1000  How to useTip: Use (CTRL + .) for SQL auto-completion.DDL and SQL commandsStart the paragraphs with the full %psql.sql prefix tag! The short notation: %psql would still be able run the queries but the syntax highlighting and the auto-completions will be disabled.You can use the standard CREATE / DROP / INSERT commands to create or modify the data model:%psql.sqldrop table if exists mytable;create table mytable (i int);insert into mytable select generate_series(1, 100);Then in a separate paragraph run the query.%psql.sqlselect * from mytable;Note: You can have multiple queries in the same paragraph but only the result from the first is displayed. [1], [2].For example, this will execute both queries but only the count result will be displayed. If you revert the order of the queries the mytable content will be shown instead.%psql.sqlselect count(*) from mytable;select * from mytable;PSQL command line toolsUse the Shell Interpreter (%sh) to access the command line PSQL interactively:%shpsql -h phd3.localdomain -U gpadmin -p 5432 <<EOF dn   qEOFThis will produce output like this:        Name        |  Owner  --------------------+--------- hawq_toolkit       | gpadmin information_schema | gpadmin madlib             | gpadmin pg_catalog         | gpadmin pg_toast           | gpadmin public             | gpadmin retail_demo        | gpadminApply Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form inside your queries. You can use both the text input and select form parametrization features%psql.sqlSELECT ${group_by}, count(*) as countFROM retail_demo.order_lineitems_pxfGROUP BY ${group_by=product_id,product_id|product_name|customer_id|store_id}ORDER BY count ${order=DESC,DESC|ASC}LIMIT ${limit=10};Example HAWQ PXF/HDFS TablesCreate HAWQ external table that read data from tab-separated-value data in HDFS.%psql.sqlCREATE EXTERNAL TABLE retail_demo.payment_methods_pxf (  payment_method_id smallint,  payment_method_code character varying(20)) LOCATION ('pxf://${NAME_NODE_HOST}:50070/retail_demo/payment_methods.tsv.gz?profile=HdfsTextSimple') FORMAT 'TEXT' (DELIMITER = E't');And retrieve content%psql.sqlselect * from retail_demo.payment_methods_pxfAuto-completionThe PSQL Interpreter provides a basic auto-completion functionality. On (Ctrl+.) it list the most relevant suggestions in a pop-up window. In addition to the SQL keyword the interpreter provides suggestions for the Schema, Table, Column names as well.",
      "url": " /interpreter/postgresql.html",
      "group": "interpreter",
      "excerpt": "Apache Zeppelin supports PostgreSQL, Apache HAWQ(incubating) and Greenplum SQL data processing engines."
    }
    ,
    
  

    "/interpreter/python.html": {
      "title": "Python 2 & 3 Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Python 2 & 3 Interpreter for Apache ZeppelinConfiguration      Property    Default    Description        zeppelin.python    python    Path of the already installed Python binary (could be python2 or python3).    If python is not in your $PATH you can set the absolute directory (example : /usr/bin/python)            zeppelin.python.maxResult    1000    Max number of dataframe rows to display.  Enabling Python InterpreterIn a notebook, to enable the Python interpreter, click on the Gear icon and select PythonUsing the Python InterpreterIn a paragraph, use %python to select the Python interpreter and then input all commands.The interpreter can only work if you already have python installed (the interpreter doesn't bring it own python binaries).To access the help, type help()Python environmentsDefaultBy default, PythonInterpreter will use python command defined in zeppelin.python property to run python process.The interpreter can use all modules already installed (with pip, easy_install...)CondaConda is an package management system and environment management system for python.%python.conda interpreter lets you change between environments.UsageList your environments%python.condaActivate an environment%python.conda activate [ENVIRONMENT_NAME]Deactivate%python.conda deactivateDocker%python.docker interpreter allows PythonInterpreter creates python process in a specified docker container.UsageActivate an environment%python.docker activate [Repository]%python.docker activate [Repository:Tag]%python.docker activate [Image Id]Deactivate%python.docker deactivateExample# activate latest tensorflow image as a python environment%python.docker activate gcr.io/tensorflow/tensorflow:latestUsing Zeppelin Dynamic FormsYou can leverage Zeppelin Dynamic Form inside your Python code.Zeppelin Dynamic Form can only be used if py4j Python library is installed in your system. If not, you can install it with pip install py4j.Example : %python### Input formprint (z.input("f1","defaultValue"))### Select formprint (z.select("f1",[("o1","1"),("o2","2")],"2"))### Checkbox formprint("".join(z.checkbox("f3", [("o1","1"), ("o2","2")],["1"])))Matplotlib integrationThe python interpreter can display matplotlib figures inline automatically using the pyplot module:%pythonimport matplotlib.pyplot as pltplt.plot([1, 2, 3])This is the recommended method for using matplotlib from within a Zeppelin notebook. The output of this command will by default be converted to HTML by implicitly making use of the %html magic. Additional configuration can be achieved using the builtin z.configure_mpl() method. For example, z.configure_mpl(width=400, height=300, fmt='svg')plt.plot([1, 2, 3])Will produce a 400x300 image in SVG format, which by default are normally 600x400 and PNG respectively. In the future, another option called angular can be used to make it possible to update a plot produced from one paragraph directly from another (the output will be %angular instead of %html). However, this feature is already available in the pyspark interpreter. More details can be found in the included "Zeppelin Tutorial: Python - matplotlib basic" tutorial notebook. If Zeppelin cannot find the matplotlib backend files (which should usually be found in $ZEPPELIN_HOME/interpreter/lib/python) in your PYTHONPATH, then the backend will automatically be set to agg, and the (otherwise deprecated) instructions below can be used for more limited inline plotting.If you are unable to load the inline backend, use z.show(plt): python%pythonimport matplotlib.pyplot as pltplt.figure()(.. ..)z.show(plt)plt.close()The z.show() function can take optional parameters to adapt graph dimensions (width and height) as well as output format (png or optionally svg).%pythonz.show(plt, width='50px')z.show(plt, height='150px', fmt='svg')Pandas integrationApache Zeppelin Table Display System provides built-in data visualization capabilities. Python interpreter leverages it to visualize Pandas DataFrames though similar z.show() API, same as with Matplotlib integration.Example:import pandas as pdrates = pd.read_csv("bank.csv", sep=";")z.show(rates)SQL over Pandas DataFramesThere is a convenience %python.sql interpreter that matches Apache Spark experience in Zeppelin and enables usage of SQL language to query Pandas DataFrames and visualization of results though built-in Table Display System.Pre-requestsPandas pip install pandasPandaSQL pip install -U pandasqlIn case default binded interpreter is Python (first in the interpreter list, under the Gear Icon), you can just use it as %sql i.efirst paragraphimport pandas as pdrates = pd.read_csv("bank.csv", sep=";")next paragraph%sqlSELECT * FROM rates WHERE age < 40Otherwise it can be referred to as %python.sqlTechnical descriptionFor in-depth technical details on current implementation please refer to python/README.md.Some features not yet implemented in the Python InterpreterInterrupt a paragraph execution (cancel() method) is currently only supported in Linux and MacOs. If interpreter runs in another operating system (for instance MS Windows) , interrupt a paragraph will close the whole interpreter. A JIRA ticket (ZEPPELIN-893) is opened to implement this feature in a next release of the interpreter.Progression bar in webUI  (getProgress() method) is currently not implemented.Code-completion is currently not implemented.",
      "url": " /interpreter/python.html",
      "group": "interpreter",
      "excerpt": "Python is a programming language that lets you work quickly and integrate systems more effectively."
    }
    ,
    
  

    "/interpreter/r.html": {
      "title": "R Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->R Interpreter for Apache ZeppelinOverviewR is a free software environment for statistical computing and graphics.To run R code and visualize plots in Apache Zeppelin, you will need R on your master node (or your dev laptop).For Centos: yum install R R-devel libcurl-devel openssl-develFor Ubuntu: apt-get install r-baseValidate your installation with a simple R command:R -e "print(1+1)"To enjoy plots, install additional libraries with:+ devtools with `R -e "install.packages('devtools', repos = 'http://cran.us.r-project.org')"`+ knitr with `R -e "install.packages('knitr', repos = 'http://cran.us.r-project.org')"`+ ggplot2 with `R -e "install.packages('ggplot2', repos = 'http://cran.us.r-project.org')"`+ Other vizualisation librairies: `R -e "install.packages(c('devtools','mplot', 'googleVis'), repos = 'http://cran.us.r-project.org'); require(devtools); install_github('ramnathv/rCharts')"`We recommend you to also install the following optional R libraries for happy data analytics:glmnetpROCdata.tablecaretsqldfwordcloudConfigurationTo run Zeppelin with the R Interpreter, the SPARK_HOME environment variable must be set. The best way to do this is by editing conf/zeppelin-env.sh.If it is not set, the R Interpreter will not be able to interface with Spark.You should also copy conf/zeppelin-site.xml.template to conf/zeppelin-site.xml. That will ensure that Zeppelin sees the R Interpreter the first time it starts up.Using the R InterpreterBy default, the R Interpreter appears as two Zeppelin Interpreters, %r and %knitr.%r will behave like an ordinary REPL.  You can execute commands as in the CLI.   R base plotting is fully supportedIf you return a data.frame, Zeppelin will attempt to display it using Zeppelin's built-in visualizations.%knitr interfaces directly against knitr, with chunk options on the first line:The two interpreters share the same environment.  If you define a variable from %r, it will be within-scope if you then make a call using knitr.Using SparkR & Moving Between LanguagesIf SPARK_HOME is set, the SparkR package will be loaded automatically:The Spark Context and SQL Context are created and injected into the local environment automatically as sc and sql.The same context are shared with the %spark, %sql and %pyspark interpreters:You can also make an ordinary R variable accessible in scala and Python:And vice versa:Caveats & TroubleshootingAlmost all issues with the R interpreter turned out to be caused by an incorrectly set SPARK_HOME.  The R interpreter must load a version of the SparkR package that matches the running version of Spark, and it does this by searching SPARK_HOME. If Zeppelin isn't configured to interface with Spark in SPARK_HOME, the R interpreter will not be able to connect to Spark.The knitr environment is persistent. If you run a chunk from Zeppelin that changes a variable, then run the same chunk again, the variable has already been changed.  Use immutable variables.(Note that %spark.r and %r are two different ways of calling the same interpreter, as are %spark.knitr and %knitr. By default, Zeppelin puts the R interpreters in the %spark. Interpreter Group.Using the %r interpreter, if you return a data.frame, HTML, or an image, it will dominate the result. So if you execute three commands, and one is hist(), all you will see is the histogram, not the results of the other commands. This is a Zeppelin limitation.If you return a data.frame (for instance, from calling head()) from the %spark.r interpreter, it will be parsed by Zeppelin's built-in data visualization system.  Why knitr Instead of rmarkdown?  Why no htmlwidgets?  In order to support htmlwidgets, which has indirect dependencies, rmarkdown uses pandoc, which requires writing to and reading from disc.  This makes it many times slower than knitr, which can operate entirely in RAM.Why no ggvis or shiny?  Supporting shiny would require integrating a reverse-proxy into Zeppelin, which is a task.Max OS X & case-insensitive filesystem.  If you try to install on a case-insensitive filesystem, which is the Mac OS X default, maven can unintentionally delete the install directory because r and R become the same subdirectory.Error unable to start device X11 with the repl interpreter.  Check your shell login scripts to see if they are adjusting the DISPLAY environment variable.  This is common on some operating systems as a workaround for ssh issues, but can interfere with R plotting.akka Library Version or TTransport errors.  This can happen if you try to run Zeppelin with a SPARK_HOME that has a version of Spark other than the one specified with -Pspark-1.x when Zeppelin was compiled.",
      "url": " /interpreter/r.html",
      "group": "interpreter",
      "excerpt": "R is a free software environment for statistical computing and graphics."
    }
    ,
    
  

    "/interpreter/scalding.html": {
      "title": "Scalding Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Scalding Interpreter for Apache ZeppelinScalding is an open source Scala library for writing MapReduce jobs.Building the Scalding InterpreterYou have to first build the Scalding interpreter by enable the scalding profile as follows:mvn clean package -Pscalding -DskipTestsEnabling the Scalding InterpreterIn a notebook, to enable the Scalding interpreter, click on the Gear icon,select Scalding, and hit Save.Configuring the InterpreterScalding interpreter runs in two modes:localhdfsIn the local mode, you can access files on the local server and scalding transformation are done locally.In hdfs mode you can access files in HDFS and scalding transformation are run as hadoop map-reduce jobs.Zeppelin comes with a pre-configured Scalding interpreter in local mode.To run the scalding interpreter in the hdfs mode you have to do the following:Set the classpath with ZEPPELIN_CLASSPATH_OVERRIDESIn conf/zeppelinenv.sh, you have to setZEPPELINCLASSPATH_OVERRIDES to the contents of 'hadoop classpath'and directories with custom jar files you need for your scalding commands.Set arguments to the scalding replThe default arguments are: "--local --repl"For hdfs mode you need to add: "--hdfs --repl"If you want to add custom jars, you need to add:"-libjars directory/:directory/"For reducer estimation, you need to add something like:"-Dscalding.reducer.estimator.classes=com.twitter.scalding.reducer_estimation.InputSizeReducerEstimator"Set max.open.instancesIf you want to control the maximum number of open interpreters, you have to select "scoped" interpreter for noteoption and set max.open.instances argument.Testing the InterpreterLocal modeIn example, by using the Alice in Wonderland tutorial, we will count words (of course!), and plot a graph of the top 10 words in the book.%scaldingimport scala.io.Source// Get the Alice in Wonderland book from gutenberg.org:val alice = Source.fromURL("http://www.gutenberg.org/files/11/11.txt").getLinesval aliceLineNum = alice.zipWithIndex.toListval alicePipe = TypedPipe.from(aliceLineNum)// Now get a list of words for the book:val aliceWords = alicePipe.flatMap { case (text, _) => text.split("s+").toList }// Now lets add a count for each word:val aliceWithCount = aliceWords.filterNot(_.equals("")).map { word => (word, 1L) }// let's sum them for each word:val wordCount = aliceWithCount.group.sumprint ("Here are the top 10 wordsn")val top10 = wordCount  .groupAll  .sortBy { case (word, count) => -count }  .take(10)top10.dump%scaldingval table = "wordst countn" + top10.toIterator.map{case (k, (word, count)) => s"$wordt$count"}.mkString("n")print("%table " + table)If you click on the icon for the pie chart, you should be able to see a chart like this:HDFS modeTest mode%scaldingmodeThis command should print:res4: com.twitter.scalding.Mode = Hdfs(true,Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml)Test HDFS readval testfile = TypedPipe.from(TextLine("/user/x/testfile"))testfile.dumpThis command should print the contents of the hdfs file /user/x/testfile.Test map-reduce jobval testfile = TypedPipe.from(TextLine("/user/x/testfile"))val a = testfile.groupAll.size.valuesa.toListThis command should create a map reduce job.Future WorkBetter user feedback (hadoop url, progress updates)Ability to cancel jobsAbility to dynamically load jars without restarting the interpreterMultiuser scalability (run scalding interpreters on different servers)",
      "url": " /interpreter/scalding.html",
      "group": "interpreter",
      "excerpt": "Scalding is an open source Scala library for writing MapReduce jobs."
    }
    ,
    
  

    "/interpreter/scio.html": {
      "title": "Scio Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Scio Interpreter for Apache ZeppelinOverviewScio is a Scala DSL for Google Cloud Dataflow and Apache Beam inspired by Spark and Scalding. See the current wiki and API documentation for more information.Configuration      Name    Default Value    Description        zeppelin.scio.argz    --runner=InProcessPipelineRunner    Scio interpreter wide arguments. Documentation: https://github.com/spotify/scio/wiki#options and https://cloud.google.com/dataflow/pipelines/specifying-exec-params        zeppelin.scio.maxResult    1000    Max number of SCollection results to display  Enabling the Scio InterpreterIn a notebook, to enable the Scio interpreter, click the Gear icon and select beam (beam.scio).Using the Scio InterpreterIn a paragraph, use %beam.scio to select the Scio interpreter. You can use it much the same way as vanilla Scala REPL and Scio REPL. State (like variables, imports, execution etc) is shared among all Scio paragraphs. There is a special variable argz which holds arguments from Scio interpreter settings. The easiest way to proceed is to create a Scio context via standard ContextAndArgs.%beam.scioval (sc, args) = ContextAndArgs(argz)Use sc context the way you would in a regular pipeline/REPL.Example:%beam.scioval (sc, args) = ContextAndArgs(argz)sc.parallelize(Seq("foo", "foo", "bar")).countByValue.closeAndDisplay()If you close Scio context, go ahead an create a new one using ContextAndArgs. Please refer to Scio wiki for more complex examples. You can close Scio context much the same way as in Scio REPL, and use Zeppelin display helpers to synchronously close and display results - read more below.ProgressThere can be only one paragraph running at once. There is no notion of overall progress, thus progress bar will show 0.SCollection display helpersScio interpreter comes with display helpers to ease working with Zeppelin notebooks. Simply use closeAndDisplay() on SCollection to close context and display the results. The number of results is limited by zeppelin.scio.maxResult (by default 1000).Supported SCollection types:Scio's typed BigQueryScala's Products (case classes, tuples)Google BigQuery's TableRowApache AvroAll Scala's AnyValHelper methodsThere are different helper methods for different objects. You can easily display results from SCollection, Future[Tap] and Tap.SCollection helperSCollection has closeAndDisplay Zeppelin helper method for types listed above. Use it to synchronously close Scio context, and once available pull and display results.Future[Tap] helperFuture[Tap] has waitAndDisplay Zeppelin helper method for types listed above. Use it to synchronously wait for results, and once available pull and display results.Tap helperTap has display Zeppelin helper method for types listed above. Use it to pull and display results.ExamplesBigQuery example:%beam.scio@BigQueryType.fromQuery("""|SELECT departure_airport,count(case when departure_delay>0 then 1 else 0 end) as no_of_delays                           |FROM [bigquery-samples:airline_ontime_data.flights]                           |group by departure_airport                           |order by 2 desc                           |limit 10""".stripMargin) class Flightsval (sc, args) = ContextAndArgs(argz)sc.bigQuerySelect(Flights.query).closeAndDisplay(Flights.schema)BigQuery typed example:%beam.scio@BigQueryType.fromQuery("""|SELECT departure_airport,count(case when departure_delay>0 then 1 else 0 end) as no_of_delays                           |FROM [bigquery-samples:airline_ontime_data.flights]                           |group by departure_airport                           |order by 2 desc                           |limit 10""".stripMargin) class Flightsval (sc, args) = ContextAndArgs(argz)sc.typedBigQuery[Flights]().flatMap(_.no_of_delays).mean.closeAndDisplay()Avro example:%beam.scioimport com.spotify.data.ExampleAvroval (sc, args) = ContextAndArgs(argz)sc.avroFile[ExampleAvro]("gs://<bucket>/tmp/my.avro").take(10).closeAndDisplay()Avro example with a view schema:%beam.scioimport com.spotify.data.ExampleAvroimport org.apache.avro.Schemaval (sc, args) = ContextAndArgs(argz)val view = Schema.parse("""{"type":"record","name":"ExampleAvro","namespace":"com.spotify.data","fields":[{"name":"track","type":"string"}, {"name":"artist", "type":"string"}]}""")sc.avroFile[EndSongCleaned]("gs://<bucket>/tmp/my.avro").take(10).closeAndDisplay(view)Google credentialsScio Interpreter will try to infer your Google Cloud credentials from its environment, it will take into the account:argz interpreter settings (doc)environment variable (GOOGLE_APPLICATION_CREDENTIALS)gcloud configurationBigQuery macro credentialsCurrently BigQuery project for macro expansion is inferred using Google Dataflow's DefaultProjectFactory().create()",
      "url": " /interpreter/scio.html",
      "group": "interpreter",
      "excerpt": "Scio is a Scala DSL for Apache Beam/Google Dataflow model."
    }
    ,
    
  

    "/interpreter/shell.html": {
      "title": "Shell interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Shell interpreter for Apache ZeppelinOverviewShell interpreter uses Apache Commons Exec to execute external processes. In Zeppelin notebook, you can use %sh in the beginning of a paragraph to invoke system shell and run commands.Note : Currently each command runs as the user Zeppelin server is running as.ConfigurationAt the "Interpreters" menu in Zeppelin dropdown menu, you can set the property value for Shell interpreter.      Name    Value    Description        shell.command.timeout.millisecs    60000    Shell command time out in millisecs        zeppelin.shell.auth.type        Types of authentications' methods supported are SIMPLE, and KERBEROS        zeppelin.shell.principal        The principal name to load from the keytab        zeppelin.shell.keytab.location        The path to the keytab file  ExampleThe following example demonstrates the basic usage of Shell in a Zeppelin notebook.If you need further information about Zeppelin Interpreter Setting for using Shell interpreter, please read What is interpreter setting? section first.",
      "url": " /interpreter/shell.html",
      "group": "interpreter",
      "excerpt": "Shell interpreter uses Apache Commons Exec to execute external processes."
    }
    ,
    
  

    "/interpreter/spark.html": {
      "title": "Apache Spark Interpreter for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Spark Interpreter for Apache ZeppelinOverviewApache Spark is a fast and general-purpose cluster computing system.It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs.Apache Spark is supported in Zeppelin with Spark interpreter group which consists of below five interpreters.      Name    Class    Description        %spark    SparkInterpreter    Creates a SparkContext and provides a Scala environment        %spark.pyspark    PySparkInterpreter    Provides a Python environment        %spark.r    SparkRInterpreter    Provides an R environment with SparkR support        %spark.sql    SparkSQLInterpreter    Provides a SQL environment        %spark.dep    DepInterpreter    Dependency loader  ConfigurationThe Spark interpreter can be configured with properties provided by Zeppelin.You can also set other Spark properties which are not listed in the table. For a list of additional properties, refer to Spark Available Properties.      Property    Default    Description        args        Spark commandline args      master    local[*]    Spark master uri.  ex) spark://masterhost:7077      spark.app.name    Zeppelin    The name of spark application.        spark.cores.max        Total number of cores to use.  Empty value uses all available core.        spark.executor.memory     1g    Executor memory per worker instance.  ex) 512m, 32g        zeppelin.dep.additionalRemoteRepository    spark-packages,  http://dl.bintray.com/spark-packages/maven,  false;    A list of id,remote-repository-URL,is-snapshot;  for each remote repository.        zeppelin.dep.localrepo    local-repo    Local repository for dependency loader        zeppelin.pyspark.python    python    Python command to run pyspark with        zeppelin.spark.concurrentSQL    false    Execute multiple SQL concurrently if set true.        zeppelin.spark.maxResult    1000    Max number of Spark SQL result to display.        zeppelin.spark.printREPLOutput    true    Print REPL output        zeppelin.spark.useHiveContext    true    Use HiveContext instead of SQLContext if it is true.        zeppelin.spark.importImplicit    true    Import implicits, UDF collection, and sql if set true.  Without any configuration, Spark interpreter works out of box in local mode. But if you want to connect to your Spark cluster, you'll need to follow below two simple steps.1. Export SPARK_HOMEIn conf/zeppelin-env.sh, export SPARK_HOME environment variable with your Spark installation path.For example,export SPARK_HOME=/usr/lib/sparkYou can optionally set more environment variables# set hadoop conf direxport HADOOP_CONF_DIR=/usr/lib/hadoop# set options to pass spark-submit commandexport SPARK_SUBMIT_OPTIONS="--packages com.databricks:spark-csv_2.10:1.2.0"# extra classpath. e.g. set classpath for hive-site.xmlexport ZEPPELIN_INTP_CLASSPATH_OVERRIDES=/etc/hive/confFor Windows, ensure you have winutils.exe in %HADOOP_HOME%bin. Please see Problems running Hadoop on Windows for the details.2. Set master in Interpreter menuAfter start Zeppelin, go to Interpreter menu and edit master property in your Spark interpreter setting. The value may vary depending on your Spark cluster deployment type.For example,local[*] in local modespark://master:7077 in standalone clusteryarn-client in Yarn client modemesos://host:5050 in Mesos clusterThat's it. Zeppelin will work with any version of Spark and any deployment type without rebuilding Zeppelin in this way. For the further information about Spark & Zeppelin version compatibility, please refer to "Available Interpreters" section in Zeppelin download page.Note that without exporting SPARK_HOME, it's running in local mode with included version of Spark. The included version may vary depending on the build profile.SparkContext, SQLContext, SparkSession, ZeppelinContextSparkContext, SQLContext and ZeppelinContext are automatically created and exposed as variable names sc, sqlContext and z, respectively, in Scala, Python and R environments.Staring from 0.6.1 SparkSession is available as variable spark when you are using Spark 2.x.Note that Scala/Python/R environment shares the same SparkContext, SQLContext and ZeppelinContext instance. Dependency ManagementThere are two ways to load external libraries in Spark interpreter. First is using interpreter setting menu and second is loading Spark properties.1. Setting Dependencies via Interpreter SettingPlease see Dependency Management for the details.2. Loading Spark PropertiesOnce SPARK_HOME is set in conf/zeppelin-env.sh, Zeppelin uses spark-submit as spark interpreter runner. spark-submit supports two ways to load configurations. The first is command line options such as --master and Zeppelin can pass these options to spark-submit by exporting SPARK_SUBMIT_OPTIONS in conf/zeppelin-env.sh. Second is reading configuration options from SPARK_HOME/conf/spark-defaults.conf. Spark properties that user can set to distribute libraries are:      spark-defaults.conf    SPARK_SUBMIT_OPTIONS    Description        spark.jars    --jars    Comma-separated list of local jars to include on the driver and executor classpaths.        spark.jars.packages    --packages    Comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. Will search the local maven repo, then maven central and any additional remote repositories given by --repositories. The format for the coordinates should be groupId:artifactId:version.        spark.files    --files    Comma-separated list of files to be placed in the working directory of each executor.  Here are few examples:SPARK_SUBMIT_OPTIONS in conf/zeppelin-env.shexport SPARK_SUBMIT_OPTIONS="--packages com.databricks:spark-csv_2.10:1.2.0 --jars /path/mylib1.jar,/path/mylib2.jar --files /path/mylib1.py,/path/mylib2.zip,/path/mylib3.egg"SPARK_HOME/conf/spark-defaults.confspark.jars        /path/mylib1.jar,/path/mylib2.jarspark.jars.packages   com.databricks:spark-csv_2.10:1.2.0spark.files       /path/mylib1.py,/path/mylib2.egg,/path/mylib3.zip3. Dynamic Dependency Loading via %spark.dep interpreterNote: %spark.dep interpreter loads libraries to %spark and %spark.pyspark but not to  %spark.sql interpreter. So we recommend you to use the first option instead.When your code requires external library, instead of doing download/copy/restart Zeppelin, you can easily do following jobs using %spark.dep interpreter.Load libraries recursively from maven repositoryLoad libraries from local filesystemAdd additional maven repositoryAutomatically add libraries to SparkCluster (You can turn off)Dep interpreter leverages Scala environment. So you can write any Scala code here.Note that %spark.dep interpreter should be used before %spark, %spark.pyspark, %spark.sql.Here's usages.%spark.depz.reset() // clean up previously added artifact and repository// add maven repositoryz.addRepo("RepoName").url("RepoURL")// add maven snapshot repositoryz.addRepo("RepoName").url("RepoURL").snapshot()// add credentials for private maven repositoryz.addRepo("RepoName").url("RepoURL").username("username").password("password")// add artifact from filesystemz.load("/path/to.jar")// add artifact from maven repository, with no dependencyz.load("groupId:artifactId:version").excludeAll()// add artifact recursivelyz.load("groupId:artifactId:version")// add artifact recursively except comma separated GroupID:ArtifactId listz.load("groupId:artifactId:version").exclude("groupId:artifactId,groupId:artifactId, ...")// exclude with patternz.load("groupId:artifactId:version").exclude(*)z.load("groupId:artifactId:version").exclude("groupId:artifactId:*")z.load("groupId:artifactId:version").exclude("groupId:*")// local() skips adding artifact to spark clusters (skipping sc.addJar())z.load("groupId:artifactId:version").local()ZeppelinContextZeppelin automatically injects ZeppelinContext as variable z in your Scala/Python environment. ZeppelinContext provides some additional functions and utilities.Object ExchangeZeppelinContext extends map and it's shared between Scala and Python environment.So you can put some objects from Scala and read it from Python, vice versa.  // Put object from scala%sparkval myObject = ...z.put("objName", myObject)// Exchanging data framesmyScalaDataFrame = ...z.put("myScalaDataFrame", myScalaDataFrame)val myPythonDataFrame = z.get("myPythonDataFrame").asInstanceOf[DataFrame]    # Get object from python%spark.pysparkmyObject = z.get("objName")# Exchanging data framesmyPythonDataFrame = ...z.put("myPythonDataFrame", postsDf._jdf)myScalaDataFrame = DataFrame(z.get("myScalaDataFrame"), sqlContext)  Form CreationZeppelinContext provides functions for creating forms.In Scala and Python environments, you can create forms programmatically.  %spark/* Create text input form */z.input("formName")/* Create text input form with default value */z.input("formName", "defaultValue")/* Create select form */z.select("formName", Seq(("option1", "option1DisplayName"),                         ("option2", "option2DisplayName")))/* Create select form with default value*/z.select("formName", "option1", Seq(("option1", "option1DisplayName"),                                    ("option2", "option2DisplayName")))    %spark.pyspark# Create text input formz.input("formName")# Create text input form with default valuez.input("formName", "defaultValue")# Create select formz.select("formName", [("option1", "option1DisplayName"),                      ("option2", "option2DisplayName")])# Create select form with default valuez.select("formName", [("option1", "option1DisplayName"),                      ("option2", "option2DisplayName")], "option1")  In sql environment, you can create form in simple template.%spark.sqlselect * from ${table=defaultTableName} where text like '%${search}%'To learn more about dynamic form, checkout Dynamic Form.Matplotlib Integration (pyspark)Both the python and pyspark interpreters have built-in support for inline visualization using matplotlib, a popular plotting library for python. More details can be found in the python interpreter documentation, since matplotlib support is identical. More advanced interactive plotting can be done with pyspark through utilizing Zeppelin's built-in Angular Display System, as shown below:Interpreter setting optionYou can choose one of shared, scoped and isolated options wheh you configure Spark interpreter. Spark interpreter creates separated Scala compiler per each notebook but share a single SparkContext in scoped mode (experimental). It creates separated SparkContext per each notebook in isolated mode.Setting up Zeppelin with KerberosLogical setup with Zeppelin, Kerberos Key Distribution Center (KDC), and Spark on YARN:Configuration SetupOn the server that Zeppelin is installed, install Kerberos client modules and configuration, krb5.conf.This is to make the server communicate with KDC.Set SPARK_HOME in [ZEPPELIN_HOME]/conf/zeppelin-env.sh to use spark-submit(Additionally, you might have to set export HADOOP_CONF_DIR=/etc/hadoop/conf)Add the two properties below to Spark configuration ([SPARK_HOME]/conf/spark-defaults.conf):spark.yarn.principalspark.yarn.keytabNOTE: If you do not have permission to access for the above spark-defaults.conf file, optionally, you can add the above lines to the Spark Interpreter setting through the Interpreter tab in the Zeppelin UI.That's it. Play with Zeppelin!",
      "url": " /interpreter/spark.html",
      "group": "interpreter",
      "excerpt": "Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution engine."
    }
    ,
    
  

    "/manual/dependencymanagement.html": {
      "title": "Dependency Management for Apache Spark Interpreter",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Dependency Management for InterpreterYou can include external libraries to interpreter by setting dependencies in interpreter menu.When your code requires external library, instead of doing download/copy/restart Zeppelin, you can easily do following jobs in this menu.Load libraries recursively from Maven repositoryLoad libraries from local filesystemAdd additional maven repositoryAutomatically add libraries to SparkCluster                         Load Dependencies to Interpreter                Click 'Interpreter' menu in navigation bar.        Click 'edit' button of the interpreter which you want to load dependencies to.        Fill artifact and exclude field to your needs.           You can enter not only groupId:artifactId:version but also local file in artifact field.        Press 'Save' to restart the interpreter with loaded libraries.                                              Add repository for dependency resolving                Press  icon in 'Interpreter' menu on the top right side.           It will show you available repository lists.       If you need to resolve dependencies from other than central maven repository or       local ~/.m2 repository, hit  icon next to repository lists.        Fill out the form and click 'Add' button, then you will be able to see that new repository is added.        Optionally, if you are behind a corporate firewall, you can specify also all proxy settings so that Zeppelin can download the dependencies using the given credentials      ",
      "url": " /manual/dependencymanagement.html",
      "group": "manual",
      "excerpt": "Include external libraries to Apache Spark Interpreter by setting dependencies in interpreter menu."
    }
    ,
    
  

    "/manual/dynamicform.html": {
      "title": "Dynamic Form in Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Dynamic FormApache Zeppelin dynamically creates input forms. Depending on language backend, there're two different ways to create dynamic form.Custom language backend can select which type of form creation it wants to use.Using form TemplatesThis mode creates form using simple template language. It's simple and easy to use. For example Markdown, Shell, Spark SQL language backend uses it.Text input formTo create text input form, use ${formName} templates.for exampleAlso you can provide default value, using ${formName=defaultValue}.Select formTo create select form, use ${formName=defaultValue,option1|option2...}for exampleAlso you can separate option's display name and value, using ${formName=defaultValue,option1(DisplayName)|option2(DisplayName)...}The paragraph will be automatically run after you change your selection by default.But in case you have multiple types dynamic form in one paragraph, you might want to run the paragraph after changing all the selections.You can control this by unchecking the below Run on selection change option in the setting menu.Even if you uncheck this option, still you can run it by pressing Enter.Checkbox formFor multi-selection, you can create a checkbox form using ${checkbox:formName=defaultValue1|defaultValue2...,option1|option2...}. The variable will be substituted by a comma-separated string based on the selected items. For example:Besides, you can specify the delimiter using ${checkbox(delimiter):formName=...}:Like select form, the paragraph will be automatically run after you change your selection by default.But in case you have multiple types of dynamic form in one paragraph, you might want to run the paragraph after changing all the selections.You can control this by unchecking the below Run on selection change option in the setting menu.Even if you uncheck this option, still you can run it by pressing Enter.Creates ProgrammaticallySome language backend uses programmatic way to create form. For example ZeppelinContext provides form creation APIHere're some examples.Text input form    %sparkprintln("Hello "+z.input("name"))        %pysparkprint("Hello "+z.input("name"))    Text input form with default value    %sparkprintln("Hello "+z.input("name", "sun"))         %pysparkprint("Hello "+z.input("name", "sun"))    Select form    %sparkprintln("Hello "+z.select("day", Seq(("1","mon"),                                    ("2","tue"),                                    ("3","wed"),                                    ("4","thurs"),                                    ("5","fri"),                                    ("6","sat"),                                    ("7","sun"))))        %pysparkprint("Hello "+z.select("day", [("1","mon"),                                ("2","tue"),                                ("3","wed"),                                ("4","thurs"),                                ("5","fri"),                                ("6","sat"),                                ("7","sun")]))    Checkbox form    %sparkval options = Seq(("apple","Apple"), ("banana","Banana"), ("orange","Orange"))println("Hello "+z.checkbox("fruit", options).mkString(" and "))        %pysparkoptions = [("apple","Apple"), ("banana","Banana"), ("orange","Orange")]print("Hello "+ " and ".join(z.checkbox("fruit", options, ["apple"])))    ",
      "url": " /manual/dynamicform.html",
      "group": "manual",
      "excerpt": "Apache Zeppelin dynamically creates input forms. Depending on language backend, there're two different ways to create dynamic form."
    }
    ,
    
  

    "/manual/dynamicinterpreterload.html": {
      "title": "Dynamic Interpreter Loading using REST API",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Dynamic Interpreter Loading using REST APIApache Zeppelin provides pluggable interpreter architecture which results in a wide and variety of the supported backend system. In this section, we will introduce Dynamic interpreter loading using REST API. This concept actually comes from Zeppelin Helium Proposal.Before we start, if you are not familiar with the concept of Zeppelin interpreter, you can check out Overview of Zeppelin interpreter first.OverviewIn the past, Zeppelin was loading interpreter binaries from /interpreter/[interpreter_name] directory. They were configured by zeppelin.interpreters property in conf/zeppelin-site.xml or ZEPPELIN_INTERPRETERS env variables in conf/zeppelin-env.sh. They were loaded on Zeppelin server startup and stayed alive until the server was stopped.In order to simplify using 3rd party interpreters, we changed this way to dynamically load interpreters from Maven Repository using REST API. Hopefully, the picture below will help you to understand the process. Load & Unload Interpreters Using REST APILoadYou can load interpreters located in Maven repository using REST API, like this:( Maybe, you are unfamiliar with [interpreter_group_name] or [interpreter_name]. If so, please checkout Interpreters in Zeppelin again. )http://[zeppelin-server]:[zeppelin-port]/api/interpreter/load/[interpreter_group_name]/[interpreter_name]The Restful method will be POST. And the parameters you need are:Artifact: Maven artifact ( groupId:artifactId:version ) Class Name: Package name + Interpreter class nameRepository ( optional ): Additional maven repository addressFor example, if you want to load markdown interpreter to your Zeppelin, the parameters and URL you need may look like:http://127.0.0.1:8080/api/interpreter/load/md/markdown{  "artifact": "org.apache.zeppelin:zeppelin-markdown:0.6.0-SNAPSHOT",  "className": "org.apache.zeppelin.markdown.Markdown",  "repository": {    "url": "http://dl.bintray.com/spark-packages/maven",    "snapshot": false  }}The meaning of each parameters is: ArtifactgroupId: org.apache.zeppelinartifactId: zeppelin-markdownversion: 0.6.0-SNAPSHOTClass NamePackage Name: org.apache.zeppelinInterpreter Class Name: markdown.MarkdownRepository ( optional )Url: http://dl.bintray.com/spark-packages/mavenSnapshot: falsePlease note: The interpreters you downloaded need to be reload, when your Zeppelin server is down. UnloadIf you want to unload the interpreters using REST API, http://[zeppelin-server]:[zeppelin-port]/api/interpreter/unload/[interpreter_group_name]/[interpreter_name]In this case, the Restful method will be DELETE.What is the next step after Loading ?Q1. Where is the location of interpreters you downloaded ?Actually, the answer about this question is in the above picture. Once the REST API is called, the .jar files of interpreters you get are saved under ZEPPELIN_HOME/local-repo first. Then, they will be copied to ZEPPELIN_HOME/interpreter directory. So, please checkout your ZEPPELIN_HOME/interpreter.Q2. Then, how can I use this interpreter ?After loading an interpreter, you can use it by creating and configuring it in Zeppelin's Interpreter tab.Oh, you don't need to restart your Zeppelin server. Because it is Dynamic Loading, you can configure and load it at runtime !After Zeppelin server up, browse Zeppelin home and click Interpreter tab.At the Interpreter section, click +Create button. Then, you can verify the interpreter list that you loaded.After choosing an interpreter, you can configure and use it. Don't forget to save it.Create a new notebook in the Notebook section, then you can bind the interpreters from your interpreter list. Just drag and drop !At last, you can use your interpreter !If you want to get the specific information about respective interpreters, please checkout each interpreter documentation. ",
      "url": " /manual/dynamicinterpreterload.html",
      "group": "manual",
      "excerpt": "Apache Zeppelin provides pluggable interpreter architecture which results in a wide and variety of the supported backend system. In this page, we will introduce dynamic interpreter loading using REST API."
    }
    ,
    
  

    "/manual/interpreterexechooks.html": {
      "title": "Interpreter Execution Hooks (Experimental)",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Interpreter Execution Hooks (Experimental)OverviewApache Zeppelin allows for users to specify additional code to be executed by an interpreter at pre and post-paragraph code execution.This is primarily useful if you need to run the same set of code for all of the paragraphs within your notebook at specific times.Currently, this feature is only available for the spark and pyspark interpreters.To specify your hook code, you may use z.registerHook(). For example, enter the following into one paragraph:%pysparkz.registerHook("post_exec", "print 'This code should be executed before the parapgraph code!'")z.registerHook("pre_exec", "print 'This code should be executed after the paragraph code!'")These calls will not take into effect until the next time you run a paragraph. In another paragraph, enter%pysparkprint "This code should be entered into the paragraph by the user!"The output should be:This code should be executed before the paragraph code!This code should be entered into the paragraph by the user!This code should be executed after the paragraph code!If you ever need to know the hook code, use z.getHook():%pysparkprint z.getHook("post_exec")print 'This code should be executed after the paragraph code!'Any call to z.registerHook() will automatically overwrite what was previously registered.To completely unregister a hook event, use z.unregisterHook(eventCode).Currently only "post_exec" and "pre_exec" are valid event codes for the Zeppelin Hook Registry system.Finally, the hook registry is internally shared by other interpreters in the same group.This would allow for hook code for one interpreter REPL to be set by another as follows:%sparkz.unregisterHook("post_exec", "pyspark")The API is identical for both the spark (scala) and pyspark (python) implementations.CaveatsCalls to z.registerHook("pre_exec", ...) should be made with care. If there are errors in your specified hook code, this will cause the interpreter REPL to become unable to execute any code pass the pre-execute stage making it impossible for direct calls to z.unregisterHook() to take into effect. Current workarounds include calling z.unregisterHook() from a different interpreter REPL in the same interpreter group (see above) or manually restarting the interpreter group in the UI. ",
      "url": " /manual/interpreterexechooks.html",
      "group": "manual",
      "excerpt": "Apache Zeppelin allows for users to specify additional code to be executed by an interpreter at pre and post-paragraph code execution."
    }
    ,
    
  

    "/manual/interpreterinstallation.html": {
      "title": "Interpreter Installation in Netinst Binary Package",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Interpreter InstallationApache Zeppelin provides Interpreter Installation mechanism for whom downloaded Zeppelin netinst binary package, or just want to install another 3rd party interpreters.Community managed interpretersApache Zeppelin provides several interpreters as community managed interpreters.If you downloaded netinst binary package, you need to install by using below commands.Install all community managed interpreters./bin/install-interpreter.sh --allInstall specific interpreters./bin/install-interpreter.sh --name md,shell,jdbc,pythonYou can get full list of community managed interpreters by running./bin/install-interpreter.sh --listInstall interpreter built with Scala 2.10Zeppelin support both Scala 2.10 and 2.11 for several interpreters as below:      Name    Maven Artifact for Scala 2.10    Maven Artifact for Scala 2.11        cassandra    org.apache.zeppelin:zeppelin-cassandra_2.10:0.7.2    org.apache.zeppelin:zeppelin-cassandra_2.11:0.7.2        flink    org.apache.zeppelin:zeppelin-flink_2.10:0.7.2    org.apache.zeppelin:zeppelin-flink_2.11:0.7.2        ignite    org.apache.zeppelin:zeppelin-ignite_2.10:0.7.2    org.apache.zeppelin:zeppelin-ignite_2.11:0.7.2        scio    org.apache.zeppelin:zeppelin-scio_2.10:0.7.2    org.apache.zeppelin:zeppelin-scio_2.11:0.7.2        spark    org.apache.zeppelin:zeppelin-spark_2.10:0.7.2    org.apache.zeppelin:zeppelin-spark_2.11:0.7.2  If you install one of these interpreters only with --name option, installer will download interpreter built with Scala 2.11 by default. If you want to specify Scala version, you will need to add --artifact option. Here is the example of installing flink interpreter built with Scala 2.10../bin/install-interpreter.sh --name flink --artifact org.apache.zeppelin:zeppelin-flink_2.10:0.7.2Install Spark interpreter built with Scala 2.10Spark distribution package has been built with Scala 2.10 until 1.6.2. If you have SPARK_HOME set pointing to Spark version earlier than 2.0.0, you need to download Spark interpreter packaged with Scala 2.10. To do so, use follow command:rm -rf ./interpreter/spark./bin/install-interpreter.sh --name spark --artifact org.apache.zeppelin:zeppelin-spark_2.10:0.7.2Once you have installed interpreters, you need to restart Zeppelin. And then create interpreter setting and bind it with your notebook.3rd party interpretersYou can also install 3rd party interpreters located in the maven repository by using below commands.Install 3rd party interpreters./bin/install-interpreter.sh --name interpreter1 --artifact groupId1:artifact1:version1The above command will download maven artifact groupId1:artifact1:version1 and all of it's transitive dependencies into interpreter/interpreter1 directory.After restart Zeppelin, then create interpreter setting and bind it with your notebook.Install multiple 3rd party interpreters at once./bin/install-interpreter.sh --name interpreter1,interpreter2 --artifact groupId1:artifact1:version1,groupId2:artifact2:version2--name and --artifact arguments will recieve comma separated list.Available community managed interpretersYou can also find the below community managed interpreter list in conf/interpreter-list file.      Name    Maven Artifact    Description        alluxio    org.apache.zeppelin:zeppelin-alluxio:0.7.2    Alluxio interpreter        angular    org.apache.zeppelin:zeppelin-angular:0.7.2    HTML and AngularJS view rendering        beam    org.apache.zeppelin:zeppelin-beam:0.7.2    Beam interpreter        bigquery    org.apache.zeppelin:zeppelin-bigquery:0.7.2    BigQuery interpreter        cassandra    org.apache.zeppelin:zeppelin-cassandra_2.11:0.7.2    Cassandra interpreter built with Scala 2.11        elasticsearch    org.apache.zeppelin:zeppelin-elasticsearch:0.7.2    Elasticsearch interpreter        file    org.apache.zeppelin:zeppelin-file:0.7.2    HDFS file interpreter        flink    org.apache.zeppelin:zeppelin-flink_2.11:0.7.2    Flink interpreter built with Scala 2.11        hbase    org.apache.zeppelin:zeppelin-hbase:0.7.2    Hbase interpreter        ignite    org.apache.zeppelin:zeppelin-ignite_2.11:0.7.2    Ignite interpreter built with Scala 2.11        jdbc    org.apache.zeppelin:zeppelin-jdbc:0.7.2    Jdbc interpreter        kylin    org.apache.zeppelin:zeppelin-kylin:0.7.2    Kylin interpreter        lens    org.apache.zeppelin:zeppelin-lens:0.7.2    Lens interpreter        livy    org.apache.zeppelin:zeppelin-livy:0.7.2    Livy interpreter        md    org.apache.zeppelin:zeppelin-markdown:0.7.2    Markdown support        pig    org.apache.zeppelin:zeppelin-pig:0.7.2    Pig interpreter        postgresql    org.apache.zeppelin:zeppelin-postgresql:0.7.2    Postgresql interpreter        python    org.apache.zeppelin:zeppelin-python:0.7.2    Python interpreter        scio    org.apache.zeppelin:zeppelin-scio_2.11:0.7.2    Scio interpreter built with Scala 2.11        shell    org.apache.zeppelin:zeppelin-shell:0.7.2    Shell command  ",
      "url": " /manual/interpreterinstallation.html",
      "group": "manual",
      "excerpt": "Apache Zeppelin provides Interpreter Installation mechanism for whom downloaded Zeppelin netinst binary package, or just want to install another 3rd party interpreters."
    }
    ,
    
  

    "/manual/interpreters.html": {
      "title": "Interpreters in Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Interpreters in Apache ZeppelinOverviewIn this section, we will explain about the role of interpreters, interpreters group and interpreter settings in Zeppelin.The concept of Zeppelin interpreter allows any language/data-processing-backend to be plugged into Zeppelin.Currently, Zeppelin supports many interpreters such as Scala ( with Apache Spark ), Python ( with Apache Spark ), Spark SQL, JDBC, Markdown, Shell and so on.What is Zeppelin interpreter?Zeppelin Interpreter is a plug-in which enables Zeppelin users to use a specific language/data-processing-backend. For example, to use Scala code in Zeppelin, you need %spark interpreter.When you click the +Create button in the interpreter page, the interpreter drop-down list box will show all the available interpreters on your server.What is interpreter setting?Zeppelin interpreter setting is the configuration of a given interpreter on Zeppelin server. For example, the properties are required for hive JDBC interpreter to connect to the Hive server.Properties are exported as environment variable when property name is consisted of upper characters, numbers and underscore ([A-Z_0-9]). Otherwise set properties as JVM property.Each notebook can be bound to multiple Interpreter Settings using setting icon on upper right corner of the notebook.What is interpreter group?Every Interpreter is belonged to an Interpreter Group. Interpreter Group is a unit of start/stop interpreter.By default, every interpreter is belonged to a single group, but the group might contain more interpreters. For example, Spark interpreter group is including Spark support, pySpark, Spark SQL and the dependency loader.Technically, Zeppelin interpreters from the same group are running in the same JVM. For more information about this, please checkout here.Each interpreters is belonged to a single group and registered together. All of their properties are listed in the interpreter setting like below image.Interpreter binding modeEach Interpreter Setting can choose one of 'shared', 'scoped', 'isolated' interpreter binding mode.In 'shared' mode, every notebook bound to the Interpreter Setting will share the single Interpreter instance. In 'scoped' mode, each notebook will create new Interpreter instance in the same interpreter process. In 'isolated' mode, each notebook will create new Interpreter process.Connecting to the existing remote interpreterZeppelin users can start interpreter thread embedded in their service. This will provide flexibility to user to start interpreter on remote host. To start interpreter along with your service you have to create an instance of RemoteInterpreterServer and start it as follows:RemoteInterpreterServer interpreter=new RemoteInterpreterServer(3678); // Here, 3678 is the port on which interpreter will listen.    interpreter.start()  The above code will start interpreter thread inside your process. Once the interpreter is started you can configure zeppelin to connect to RemoteInterpreter by checking Connect to existing process checkbox and then provide Host and Port on which interpreter process is listening as shown in the image below:",
      "url": " /manual/interpreters.html",
      "group": "manual",
      "excerpt": "This document explains about the role of interpreters, interpreters group and interpreter settings in Apache Zeppelin. The concept of Zeppelin interpreter allows any language/data-processing-backend to be plugged into Zeppelin."
    }
    ,
    
  

    "/manual/notebookashomepage.html": {
      "title": "Customize Apache Zeppelin homepage",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Customize Apache Zeppelin homepageApache Zeppelin allows you to use one of the notes you create as your Zeppelin Homepage.With that you can brand your Zeppelin installation, adjust the instruction to your users needs and even translate to other languages.How to set a note as your Zeppelin homepageThe process for creating your homepage is very simple as shown below:Create a note using ZeppelinSet the note id in the config fileRestart ZeppelinCreate a note using ZeppelinCreate a new note using Zeppelin,you can use %md interpreter for markdown content or any other interpreter you like.You can also use the display system to generate text, html, table orAngular (backend API, frontend API).Run (shift+Enter) the note and see the output. Optionally, change the note view to report to hidethe code sections.Set the note id in the config fileTo set the note id in the config file, you should copy it from the last word in the note url.For example,Set the note id to the ZEPPELIN_NOTEBOOK_HOMESCREEN environment variableor zeppelin.notebook.homescreen property.You can also set the ZEPPELIN_NOTEBOOK_HOMESCREEN_HIDE environment variableor zeppelin.notebook.homescreen.hide property to hide the new note from the note list.Restart ZeppelinRestart your Zeppelin server./bin/zeppelin-daemon stop./bin/zeppelin-daemon startThat's it! Open your browser and navigate to Apache Zeppelin and see your customized homepage.Show note list in your custom homepageIf you want to display the list of notes on your custom Apache Zeppelin homepage allyou need to do is use our %angular support.Add the following code to a paragraph in your Apache Zeppelin note and run it.%sparkprintln("""%angular  <div ng-include="'app/home/notebook.html'"></div>""")After running the paragraph, you will see output similar to this one:That's it! Voila! You have your note list.",
      "url": " /manual/notebookashomepage.html",
      "group": "manual",
      "excerpt": "Apache Zeppelin allows you to use one of the notes you create as your Zeppelin Homepage. With that you can brand your Zeppelin installation, adjust the instruction to your users needs and even translate to other languages."
    }
    ,
    
  

    "/manual/publish.html": {
      "title": "How can you publish your paragraph",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->How can you publish your paragraph ?Apache Zeppelin provides a feature for publishing your notebook paragraph results. Using this feature, you can show Zeppelin notebook paragraph results in your own website. It's very straightforward. Just use <iframe> tag in your page.Copy a Paragraph LinkA first step to publish your paragraph result is Copy a Paragraph Link.After running a paragraph in your Zeppelin notebook, click a gear button located on the right side. Then, click Link this Paragraph menu like below image.Just copy the provided link. Embed the Paragraph to Your WebsiteFor publishing the copied paragraph, you may use <iframe> tag in your website page.For example,<iframe src="http://< ip-address >:< port >/#/notebook/2B3QSZTKR/paragraph/...?asIframe" height="" width="" ></iframe>Finally, you can show off your beautiful visualization results in your website. Note: To embed the paragraph in a website, Apache Zeppelin needs to be reachable by that website. And please use this feature with caution and in a trusted environment only, as Zeppelin entire Webapp could be accessible by whoever visits your website. ",
      "url": " /manual/publish.html",
      "group": "manual",
      "excerpt": "Apache Zeppelin provides a feature for publishing your notebook paragraph results. Using this feature, you can show Zeppelin notebook paragraph results in your own website."
    }
    ,
    
  

    "/manual/userimpersonation.html": {
      "title": "Run zeppelin interpreter process as web front end user",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Run zeppelin interpreter process as web front end userEnable shiro auth in shiro.ini[users]user1 = password1, role1user2 = password2, role2Enable password-less ssh for the user you want to impersonate (say user1).adduser user1#ssh-keygen (optional if you don't already have generated ssh-key.ssh user1@localhost mkdir -p .sshcat ~/.ssh/id_rsa.pub | ssh user1@localhost 'cat >> .ssh/authorized_keys'Alternatively instead of password-less, user can override ZEPPELINIMPERSONATECMD in zeppelin-env.shexport ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c 'Start zeppelin server.            Screenshot                                    Go to interpreter setting page, and enable "User Impersonate" in any of the interpreter (in my example its shell interpreter)Test with a simple paragraph%shwhoamiNote that usage of "User Impersonate" option will enable Spark interpreter to use --proxy-user option with current user by default. If you want to disable --proxy-user option, then refer to ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER variable in conf/zeppelin-env.sh",
      "url": " /manual/userimpersonation.html",
      "group": "manual",
      "excerpt": "Set up zeppelin interpreter process as web front end user."
    }
    ,
    
  
  

    "/quickstart/explorezeppelinui.html": {
      "title": "Explore Apache Zeppelin UI",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Explore Apache Zeppelin UIMain homeThe first time you connect to Zeppelin, you'll land at the main page similar to the below screen capture.On the left of the page are listed all existing notes. Those notes are stored by default in the $ZEPPELIN_HOME/notebook folder.You can filter them by name using the input text form. You can also create a new note, refresh the list of existing notes(in case you manually copy them into the $ZEPPELIN_HOME/notebook folder) and import a note.When clicking on Import Note link, a new dialog open. From there you can import your note from local disk or from a remote locationif you provide the URL.By default, the name of the imported note is the same as the original note but you can override it by providing a new name.MenusNotebookThe Notebook menu proposes almost the same features as the note management section in the home page. From the drop-down menu you can:Open a selected noteFilter node by nameCreate a new noteSettingsThis menu gives you access to settings and displays information about Zeppelin. User name is set to anonymous if you use default shiro configuration. If you want to set up authentification, see Shiro authentication.About ZeppelinYou can check Zeppelin version in this menu.InterpreterIn this menu you can:Configure existing interpreter instanceAdd/remove interpreter instancesCredentialThis menu allows you to save credentials for data sources which are passed to interpreters.ConfigurationThis menu displays all the Zeppelin configuration that are set in the config file $ZEPPELIN_HOME/conf/zeppelin-site.xmlNote LayoutEach Zeppelin note is composed of 1 .. N paragraphs. The note can be viewed as a paragraph container.ParagraphEach paragraph consists of 2 sections: code section where you put your source code and result section where you can see the result of the code execution.On the top-right corner of each paragraph there are some commands to:execute the paragraph codehide/show code sectionhide/show result sectionconfigure the paragraphTo configure the paragraph, just click on the gear icon:From this dialog, you can (in descending order):find the paragraph id ( 20150924-163507_134879501 )control paragraph width. Since Zeppelin is using the grid system of Twitter Bootstrap, each paragraph width can be changed from 1 to 12move the paragraph 1 level upmove the paragraph 1 level downcreate a new paragraphchange paragraph titleshow/hide line number in the code sectiondisable the run button for this paragraphexport the current paragraph as an iframe and open the iframe in a new windowclear the result sectiondelete the current paragraphNote toolbarAt the top of the note, you can find a toolbar which exposes command buttons as well as configuration, security and display options.On the far right is displayed the note name, just click on it to reveal the input form and update it.In the middle of the toolbar you can find the command buttons:execute all the paragraphs sequentially, in their display orderhide/show code section of all paragraphshide/show result section of all paragraphsclear the result section of all paragraphsclone the current noteexport the current note to a JSON file. _Please note that the code section and result section of all paragraphs will be exported. If you have heavy data in the result section of some paragraphs, it is recommended to clean them before exportingcommit the current node contentdelete the noteschedule the execution of all paragraph using a CRON syntaxOn the right of the note tool bar you can find configuration icons:display all the keyboard shorcutsconfigure the interpreters binding to the current noteconfigure the note permissionsswitch the node display mode between default, simple and report",
      "url": " /quickstart/explorezeppelinui.html",
      "group": "quickstart",
      "excerpt": "If you are new to Apache Zeppelin, this document will guide you about the basic components of Zeppelin one by one."
    }
    ,
    
  

    "/quickstart/install_with_flink_and_spark_cluster.html": {
      "title": "Install Zeppelin with Flink and Spark in cluster mode",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Install with flink and spark clusterThis tutorial is extremely entry-level. It assumes no prior knowledge of Linux, git, or other tools. If you carefully type what I tell you when I tell you, you should be able to get Zeppelin running.Installing Zeppelin with Flink and Spark in cluster modeThis tutorial assumes the user has a machine (real or virtual with a fresh, minimal installation of Ubuntu 14.04.3 Server.Note: On the size requirements of the Virtual Machine, some users reported trouble when using the default virtual machine sizes, specifically that the hard drive needed to be at least 16GB- other users did not have this issue.There are many good tutorials on how to install Ubuntu Server on a virtual box, here is one of themRequired ProgramsAssuming the minimal install, there are several programs that we will need to install before Zeppelin, Flink, and Spark.gitopenssh-serverOpenJDK 7Maven 3.1+For git, openssh-server, and OpenJDK 7 we will be using the apt package manager.gitFrom the command prompt:sudo apt-get install gitopenssh-serversudo apt-get install openssh-serverOpenJDK 7sudo apt-get install openjdk-7-jdk openjdk-7-jre-libA note for those using Ubuntu 16.04: To install openjdk-7 on Ubuntu 16.04, one must add a repository.  Sourcesudo add-apt-repository ppa:openjdk-r/ppasudo apt-get updatesudo apt-get install openjdk-7-jdk openjdk-7-jre-libMaven 3.1+Zeppelin requires maven version 3.x.  The version available in the repositories at the time of writing is 2.x, so maven must be installed manually.Purge any existing versions of maven.sudo apt-get purge maven maven2Download the maven 3.3.9 binary.wget "http://www.us.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz"Unarchive the binary and move to the /usr/local directory.tar -zxvf apache-maven-3.3.9-bin.tar.gzsudo mv ./apache-maven-3.3.9 /usr/localCreate symbolic links in /usr/bin.sudo ln -s /usr/local/apache-maven-3.3.9/bin/mvn /usr/bin/mvnInstalling ZeppelinThis provides a quick overview of Zeppelin installation from source, however the reader is encouraged to review the Zeppelin Installation GuideFrom the command prompt:Clone Zeppelin.git clone https://github.com/apache/zeppelin.gitEnter the Zeppelin root directory.cd zeppelinPackage Zeppelin.mvn clean package -DskipTests -Pspark-1.6 -Dflink.version=1.1.3 -Pscala-2.10-DskipTests skips build tests- you're not developing (yet), so you don't need to do tests, the clone version should build.-Pspark-1.6 tells maven to build a Zeppelin with Spark 1.6.  This is important because Zeppelin has its own Spark interpreter and the versions must be the same.-Dflink.version=1.1.3 tells maven specifically to build Zeppelin with Flink version 1.1.3.--Pscala-2.10 tells maven to build with Scala v2.10.Note: You may wish to include additional build flags such as -Ppyspark or -Psparkr.  See the build section of github for more details.Note: You can build against any version of Spark that has a Zeppelin build profile available. The key is to make sure you check out the matching version of Spark to build. At the time of this writing, Spark 1.6 was the most recent Spark version available.Note: On build failures. Having installed Zeppelin close to 30 times now, I will tell you that sometimes the build fails for seemingly no reason.As long as you didn't edit any code, it is unlikely the build is failing because of something you did. What does tend to happen, is some dependency that maven is trying to download is unreachable.  If your build fails on this step here are some tips:- Don't get discouraged.- Scroll up and read through the logs. There will be clues there.- Retry (that is, run the mvn clean package -DskipTests -Pspark-1.6 again)- If there were clues that a dependency couldn't be downloaded wait a few hours or even days and retry again. Open source software when compiling is trying to download all of the dependencies it needs, if a server is off-line there is nothing you can do but wait for it to come back.- Make sure you followed all of the steps carefully.- Ask the community to help you. Go here and join the user mailing list. People are there to help you. Make sure to copy and paste the build output (everything that happened in the console) and include that in your message.Start the Zeppelin daemon.bin/zeppelin-daemon.sh startUse ifconfig to determine the host machine's IP address. If you are not familiar with how to do this, a fairly comprehensive post can be found here.Open a web-browser on a machine connected to the same network as the host (or in the host operating system if using a virtual machine).  Navigate to http://yourip:8080, where yourip is the IP address you found in ifconfig.See the Zeppelin tutorial for basic Zeppelin usage. It is also advised that you take a moment to check out the tutorial notebook that is included with each Zeppelin install, and to familiarize yourself with basic notebook functionality.Flink TestCreate a new notebook named "Flink Test" and copy and paste the following code.%flink  // let Zeppelin know what interpreter to use.val text = benv.fromElements("In the time of chimpanzees, I was a monkey",   // some lines of text to analyze"Butane in my veins and I'm out to cut the junkie","With the plastic eyeballs, spray paint the vegetables","Dog food stalls with the beefcake pantyhose","Kill the headlights and put it in neutral","Stock car flamin' with a loser in the cruise control","Baby's in Reno with the Vitamin D","Got a couple of couches, sleep on the love seat","Someone came in sayin' I'm insane to complain","About a shotgun wedding and a stain on my shirt","Don't believe everything that you breathe","You get a parking violation and a maggot on your sleeve","So shave your face with some mace in the dark","Savin' all your food stamps and burnin' down the trailer park","Yo, cut it")/*  The meat and potatoes:        this tells Flink to iterate through the elements, in this case strings,        transform the string to lower case and split the string at white space into individual words        then finally aggregate the occurrence of each word.        This creates the count variable which is a list of tuples of the form (word, occurances)counts.collect().foreach(println(_))  // execute the script and print each element in the counts list*/val counts = text.flatMap{ _.toLowerCase.split("W+") }.map { (_,1) }.groupBy(0).sum(1)counts.collect().foreach(println(_))  // execute the script and print each element in the counts listRun the code to make sure the built-in Zeppelin Flink interpreter is working properly.Spark TestCreate a new notebook named "Spark Test" and copy and paste the following code.%spark // let Zeppelin know what interpreter to use.val text = sc.parallelize(List("In the time of chimpanzees, I was a monkey",  // some lines of text to analyze"Butane in my veins and I'm out to cut the junkie","With the plastic eyeballs, spray paint the vegetables","Dog food stalls with the beefcake pantyhose","Kill the headlights and put it in neutral","Stock car flamin' with a loser in the cruise control","Baby's in Reno with the Vitamin D","Got a couple of couches, sleep on the love seat","Someone came in sayin' I'm insane to complain","About a shotgun wedding and a stain on my shirt","Don't believe everything that you breathe","You get a parking violation and a maggot on your sleeve","So shave your face with some mace in the dark","Savin' all your food stamps and burnin' down the trailer park","Yo, cut it"))/*  The meat and potatoes:        this tells spark to iterate through the elements, in this case strings,        transform the string to lower case and split the string at white space into individual words        then finally aggregate the occurrence of each word.        This creates the count variable which is a list of tuples of the form (word, occurances)*/val counts = text.flatMap { _.toLowerCase.split("W+") }                 .map { (_,1) }                 .reduceByKey(_ + _)counts.collect().foreach(println(_))  // execute the script and print each element in the counts listRun the code to make sure the built-in Zeppelin Flink interpreter is working properly.Finally, stop the Zeppelin daemon.  From the command prompt run:bin/zeppelin-daemon.sh stopInstalling ClustersFlink ClusterDownload BinariesBuilding from source is recommended  where possible, for simplicity in this tutorial we will download Flink and Spark Binaries.To download the Flink Binary use wgetwget "http://mirror.cogentco.com/pub/apache/flink/flink-1.1.3/flink-1.1.3-bin-hadoop24-scala_2.10.tgz"tar -xzvf flink-1.1.3-bin-hadoop24-scala_2.10.tgzThis will download Flink 1.1.3, compatible with Hadoop 2.4.  You do not have to install Hadoop for this binary to work, but if you are using Hadoop, please change 24 to your appropriate version.Start the Flink Cluster.flink-1.1.3/bin/start-cluster.shBuilding From sourceIf you wish to build Flink from source, the following will be instructive.  Note that if you have downloaded and used the binary version this should be skipped.  The changing nature of build tools and versions across platforms makes this section somewhat precarious.  For example, Java8 and Maven 3.0.3 are recommended for building Flink, which are not recommended for Zeppelin at the time of writing.  If the user wishes to attempt to build from source, this section will provide some reference.  If errors are encountered, please contact the Apache Flink community.See the Flink Installation guide for more detailed instructions.Return to the directory where you have been downloading, this tutorial assumes that is $HOME. Clone Flink,  check out release-1.1.3-rc2, and build.cd $HOMEgit clone https://github.com/apache/flink.gitcd flinkgit checkout release-1.1.3-rc2mvn clean install -DskipTestsStart the Flink Cluster in stand-alone modebuild-target/bin/start-cluster.shEnsure the cluster is upIn a browser, navigate to http://yourip:8082 to see the Flink Web-UI.  Click on 'Task Managers' in the left navigation bar. Ensure there is at least one Task Manager present.If no task managers are present, restart the Flink cluster with the following commands:(if binaries)flink-1.1.3/bin/stop-cluster.shflink-1.1.3/bin/start-cluster.sh(if built from source)build-target/bin/stop-cluster.shbuild-target/bin/start-cluster.shSpark 1.6 ClusterDownload BinariesBuilding from source is recommended  where possible, for simplicity in this tutorial we will download Flink and Spark Binaries.Using binaries is alsoTo download the Spark Binary use wgetwget "http://d3kbcqa49mib13.cloudfront.net/spark-1.6.3-bin-hadoop2.6.tgz"tar -xzvf spark-1.6.3-bin-hadoop2.6.tgzmv spark-1.6.3-bin-hadoop2.6 sparkThis will download Spark 1.6.3, compatible with Hadoop 2.6.  You do not have to install Hadoop for this binary to work, but if you are using Hadoop, please change 2.6 to your appropriate version.Building From sourceSpark is an extraordinarily large project, which takes considerable time to download and build. It is also prone to build failures for similar reasons listed in the Flink section.  If the user wishes to attempt to build from source, this section will provide some reference.  If errors are encountered, please contact the Apache Spark community.See the Spark Installation guide for more detailed instructions.Return to the directory where you have been downloading, this tutorial assumes that is $HOME. Clone Spark, check out branch-1.6, and build.Note: Recall, we're only checking out 1.6 because it is the most recent Spark for which a Zeppelin profile exists at  the time of writing. You are free to check out other version, just make sure you build Zeppelin against the correct version of Spark. However if you use Spark 2.0, the word count example will need to be changed as Spark 2.0 is not compatible with the following examples.cd $HOMEClone, check out, and build Spark version 1.6.x.git clone https://github.com/apache/spark.gitcd sparkgit checkout branch-1.6mvn clean package -DskipTestsStart the Spark clusterReturn to the $HOME directory.cd $HOMEStart the Spark cluster in stand alone mode, specifying the webui-port as some port other than 8080 (the webui-port of Zeppelin).spark/sbin/start-master.sh --webui-port 8082Note: Why --webui-port 8082? There is a digression toward the end of this document that explains this.Open a browser and navigate to http://yourip:8082 to ensure the Spark master is running.Toward the top of the page there will be a URL: spark://yourhost:7077.  Note this URL, the Spark Master URI, it will be needed in subsequent steps.Start the slave using the URI from the Spark master WebUI:spark/sbin/start-slave.sh spark://yourhostname:7077Return to the root directory and start the Zeppelin daemon.cd $HOMEzeppelin/bin/zeppelin-daemon.sh startConfigure InterpretersOpen a web browser and go to the Zeppelin web-ui at http://yourip:8080.Now go back to the Zeppelin web-ui at http://yourip:8080 and this time click on anonymous at the top right, which will open a drop-down menu, select Interpreters to enter interpreter configuration.In the Spark section, click the edit button in the top right corner to make the property values editable (looks like a pencil).The only field that needs to be edited in the Spark interpreter is the master field. Change this value from local[*] to the URL you used to start the slave, mine was spark://ubuntu:7077.Click Save to update the parameters, and click OK when it asks you about restarting the interpreter.Now scroll down to the Flink section. Click the edit button and change the value of host from local to localhost. Click Save again.Reopen the examples and execute them again (I.e. you need to click the play button at the top of the screen, or the button on the paragraph .You should be able check the Flink and Spark webuis (at something like http://yourip:8081, http://yourip:8082, http://yourip:8083) and see that jobs have been run against the clusters.Digression Sorry to be vague and use terms such as 'something like', but exactly what web-ui is at what port is going to depend on what order you started things. What is really going on here is you are pointing your browser at specific ports, namely 8081, 8082, and 8083.  Flink and Spark all want to put their web-ui on port 8080, but are well behaved and will take the next port available. Since Zeppelin started first, it will get port 8080.  When Flink starts (assuming you started Flink first), it will try to bind to port 8080, see that it is already taken, and go to the next one available, hopefully 8081.  Spark has a webui for the master and the slave, so when they start they will try to bind to 8080   already taken by Zeppelin), then 8081 (already taken by Flink's webui), then 8082. If everything goes smoothy and you followed the directions precisely, the webuis should be 8081 and 8082.     It is possible to specify the port you want the webui to bind to (at the command line by passing the --webui-port <port> flag when you start the Flink and Spark, where <port> is the port     you want to see that webui on.  You can also set the default webui port of Spark and Flink (and Zeppelin) in the configuration files, but this is a tutorial for novices and slightly out of scope.Next StepsCheck out the tutorial for more cool things you can do with your new toy!Join the community, ask questions and contribute! Every little bit helps.",
      "url": " /quickstart/install_with_flink_and_spark_cluster.html",
      "group": "tutorial",
      "excerpt": "Tutorial is valid for Spark 1.6.x and Flink 1.1.2"
    }
    ,
    
  

    "/quickstart/tutorial.html": {
      "title": "Apache Zeppelin Tutorial",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Zeppelin TutorialThis tutorial walks you through some of the fundamental Zeppelin concepts. We will assume you have already installed Zeppelin. If not, please see here first.Current main backend processing engine of Zeppelin is Apache Spark. If you're new to this system, you might want to start by getting an idea of how it processes data to get the most out of Zeppelin.Tutorial with Local FileData RefineBefore you start Zeppelin tutorial, you will need to download bank.zip. First, to transform csv format data into RDD of Bank objects, run following script. This will also remove header using filter function.val bankText = sc.textFile("yourPath/bank/bank-full.csv")case class Bank(age:Integer, job:String, marital : String, education : String, balance : Integer)// split each line, filter out header (starts with "age"), and map it into Bank case classval bank = bankText.map(s=>s.split(";")).filter(s=>s(0)!=""age"").map(    s=>Bank(s(0).toInt,             s(1).replaceAll(""", ""),            s(2).replaceAll(""", ""),            s(3).replaceAll(""", ""),            s(5).replaceAll(""", "").toInt        ))// convert to DataFrame and create temporal tablebank.toDF().registerTempTable("bank")Data RetrievalSuppose we want to see age distribution from bank. To do this, run:%sql select age, count(1) from bank where age < 30 group by age order by ageYou can make input box for setting age condition by replacing 30 with ${maxAge=30}.%sql select age, count(1) from bank where age < ${maxAge=30} group by age order by ageNow we want to see age distribution with certain marital status and add combo box to select marital status. Run:%sql select age, count(1) from bank where marital="${marital=single,single|divorced|married}" group by age order by ageTutorial with Streaming DataData RefineSince this tutorial is based on Twitter's sample tweet stream, you must configure authentication with a Twitter account. To do this, take a look at Twitter Credential Setup. After you get API keys, you should fill out credential related values(apiKey, apiSecret, accessToken, accessTokenSecret) with your API keys on following script.This will create a RDD of Tweet objects and register these stream data as a table:import org.apache.spark.streaming._import org.apache.spark.streaming.twitter._import org.apache.spark.storage.StorageLevelimport scala.io.Sourceimport scala.collection.mutable.HashMapimport java.io.Fileimport org.apache.log4j.Loggerimport org.apache.log4j.Levelimport sys.process.stringSeqToProcess/** Configures the Oauth Credentials for accessing Twitter */def configureTwitterCredentials(apiKey: String, apiSecret: String, accessToken: String, accessTokenSecret: String) {  val configs = new HashMap[String, String] ++= Seq(    "apiKey" -> apiKey, "apiSecret" -> apiSecret, "accessToken" -> accessToken, "accessTokenSecret" -> accessTokenSecret)  println("Configuring Twitter OAuth")  configs.foreach{ case(key, value) =>    if (value.trim.isEmpty) {      throw new Exception("Error setting authentication - value for " + key + " not set")    }    val fullKey = "twitter4j.oauth." + key.replace("api", "consumer")    System.setProperty(fullKey, value.trim)    println("tProperty " + fullKey + " set as [" + value.trim + "]")  }  println()}// Configure Twitter credentialsval apiKey = "xxxxxxxxxxxxxxxxxxxxxxxxx"val apiSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"val accessToken = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"val accessTokenSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"configureTwitterCredentials(apiKey, apiSecret, accessToken, accessTokenSecret)import org.apache.spark.streaming.twitter._val ssc = new StreamingContext(sc, Seconds(2))val tweets = TwitterUtils.createStream(ssc, None)val twt = tweets.window(Seconds(60))case class Tweet(createdAt:Long, text:String)twt.map(status=>  Tweet(status.getCreatedAt().getTime()/1000, status.getText())).foreachRDD(rdd=>  // Below line works only in spark 1.3.0.  // For spark 1.1.x and spark 1.2.x,  // use rdd.registerTempTable("tweets") instead.  rdd.toDF().registerAsTable("tweets"))twt.printssc.start()Data RetrievalFor each following script, every time you click run button you will see different result since it is based on real-time data.Let's begin by extracting maximum 10 tweets which contain the word girl.%sql select * from tweets where text like '%girl%' limit 10This time suppose we want to see how many tweets have been created per sec during last 60 sec. To do this, run:%sql select createdAt, count(1) from tweets group by createdAt order by createdAtYou can make user-defined function and use it in Spark SQL. Let's try it by making function named sentiment. This function will return one of the three attitudes( positive, negative, neutral ) towards the parameter.def sentiment(s:String) : String = {    val positive = Array("like", "love", "good", "great", "happy", "cool", "the", "one", "that")    val negative = Array("hate", "bad", "stupid", "is")    var st = 0;    val words = s.split(" ")        positive.foreach(p =>        words.foreach(w =>            if(p==w) st = st+1        )    )    negative.foreach(p=>        words.foreach(w=>            if(p==w) st = st-1        )    )    if(st>0)        "positivie"    else if(st<0)        "negative"    else        "neutral"}// Below line works only in spark 1.3.0.// For spark 1.1.x and spark 1.2.x,// use sqlc.registerFunction("sentiment", sentiment _) instead.sqlc.udf.register("sentiment", sentiment _)To check how people think about girls using sentiment function we've made above, run this:%sql select sentiment(text), count(1) from tweets where text like '%girl%' group by sentiment(text)",
      "url": " /quickstart/tutorial.html",
      "group": "quickstart",
      "excerpt": "This tutorial page contains a short walk-through tutorial that uses Apache Spark backend. Please note that this tutorial is valid for Spark 1.3 and higher."
    }
    ,
    
  

    "/rest-api/rest-configuration.html": {
      "title": "Apache Zeppelin Configuration REST API",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Zeppelin Configuration REST APIOverviewApache Zeppelin provides several REST APIs for interaction and remote activation of zeppelin functionality.All REST APIs are available starting with the following endpoint http://[zeppelin-server]:[zeppelin-port]/api. Note that Apache Zeppelin REST APIs receive or return JSON objects, it is recommended for you to install some JSON viewers such as JSONView.If you work with Apache Zeppelin and find a need for an additional REST API, please file an issue or send us an email.nd a need for an additional REST API, please file an issue or send us mail.Configuration REST API listList all key/value pair of configurations              Description      This GET method return all key/value pair of configurations on the server.       Note: For security reason, some pairs would not be shown.              URL      http://[zeppelin-server]:[zeppelin-port]/api/configurations/all              Success code      200               Fail code       500                sample JSON response                    {  "status": "OK",  "message": "",  "body": {    "zeppelin.war.tempdir": "webapps",    "zeppelin.notebook.homescreen.hide": "false",    "zeppelin.interpreter.remoterunner": "bin/interpreter.sh",    "zeppelin.notebook.s3.user": "user",    "zeppelin.server.port": "8089",    "zeppelin.dep.localrepo": "local-repo",    "zeppelin.ssl.truststore.type": "JKS",    "zeppelin.ssl.keystore.path": "keystore",    "zeppelin.notebook.s3.bucket": "zeppelin",    "zeppelin.server.addr": "0.0.0.0",    "zeppelin.ssl.client.auth": "false",    "zeppelin.server.context.path": "/",    "zeppelin.ssl.keystore.type": "JKS",    "zeppelin.ssl.truststore.path": "truststore",    "zeppelin.interpreters": "org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkRInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.flink.FlinkInterpreter,org.apache.zeppelin.lens.LensInterpreter,org.apache.zeppelin.ignite.IgniteInterpreter,org.apache.zeppelin.ignite.IgniteSqlInterpreter,org.apache.zeppelin.cassandra.CassandraInterpreter,org.apache.zeppelin.geode.GeodeOqlInterpreter,org.apache.zeppelin.postgresql.PostgreSqlInterpreter,org.apache.zeppelin.kylin.KylinInterpreter,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter,org.apache.zeppelin.scalding.ScaldingInterpreter",    "zeppelin.ssl": "false",    "zeppelin.notebook.autoInterpreterBinding": "true",    "zeppelin.notebook.homescreen": "",    "zeppelin.notebook.storage": "org.apache.zeppelin.notebook.repo.VFSNotebookRepo",    "zeppelin.interpreter.connect.timeout": "30000",    "zeppelin.anonymous.allowed": "true",    "zeppelin.server.allowed.origins":"*",    "zeppelin.encoding": "UTF-8"  }}      List all prefix matched key/value pair of configurations              Description      This GET method return all prefix matched key/value pair of configurations on the server.      Note: For security reason, some pairs would not be shown.              URL      http://[zeppelin-server]:[zeppelin-port]/api/configurations/prefix/[prefix]              Success code      200               Fail code       500                sample JSON response            {  "status": "OK",  "message": "",  "body": {    "zeppelin.ssl.keystore.type": "JKS",    "zeppelin.ssl.truststore.path": "truststore",    "zeppelin.ssl.truststore.type": "JKS",    "zeppelin.ssl.keystore.path": "keystore",    "zeppelin.ssl": "false",    "zeppelin.ssl.client.auth": "false"  }}            ",
      "url": " /rest-api/rest-configuration.html",
      "group": "rest-api",
      "excerpt": "This page contains Apache Zeppelin Configuration REST API information."
    }
    ,
    
  

    "/rest-api/rest-credential.html": {
      "title": "Apache Zeppelin Credential REST API",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Zeppelin Credential REST APIOverviewApache Zeppelin provides several REST APIs for interaction and remote activation of zeppelin functionality.All REST APIs are available starting with the following endpoint http://[zeppelin-server]:[zeppelin-port]/api. Note that Apache Zeppelin REST APIs receive or return JSON objects, it is recommended for you to install some JSON viewers such as JSONView.If you work with Apache Zeppelin and find a need for an additional REST API, please file an issue or send us an email.Credential REST API ListList Credential information              Description      This GET method returns all key/value pairs of the credential information on the server.              URL      http://[zeppelin-server]:[zeppelin-port]/api/credential              Success code      200               Fail code       500                sample JSON response                    {  "status": "OK",  "message": "",  "body": {    "userCredentials":{      "entity1":{        "username":"user1",        "password":"password1"      },      "entity2":{        "username":"user2",        "password":"password2"      }    }  }}      Create an Credential Information              Description      This PUT method creates the credential information with new properties.              URL      http://[zeppelin-server]:[zeppelin-port]/api/credential/              Success code      200              Fail code       500               Sample JSON input              {  "entity": "e1",  "username": "user",  "password": "password"}                            Sample JSON response              {  "status": "OK"}                    Delete all Credential Information              Description      This DELETE method deletes the credential information.              URL      http://[zeppelin-server]:[zeppelin-port]/api/credential              Success code      200               Fail code       500               Sample JSON response              {"status":"OK"}            Delete an Credential entity              Description      This DELETE method deletes a given credential entity.              URL      http://[zeppelin-server]:[zeppelin-port]/api/credential/[entity]              Success code      200               Fail code       500               Sample JSON response              {"status":"OK"}            ",
      "url": " /rest-api/rest-credential.html",
      "group": "rest-api",
      "excerpt": "This page contains Apache Zeppelin Credential REST API information."
    }
    ,
    
  

    "/rest-api/rest-helium.html": {
      "title": "Apache Zeppelin Helium REST API",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Zeppelin Helium REST APIOverviewApache Zeppelin provides several REST APIs for interaction and remote activation of zeppelin functionality.All REST APIs are available starting with the following endpoint http://[zeppelin-server]:[zeppelin-port]/api. Note that Apache Zeppelin REST APIs receive or return JSON objects, it is recommended for you to install some JSON viewers such as JSONView.If you work with Apache Zeppelin and find a need for an additional REST API, please file an issue or send us an email.Helium REST API ListList of all available helium packages              Description      This GET method returns all the available helium packages in configured registries.              URL      http://[zeppelin-server]:[zeppelin-port]/api/helium/all              Success code      200              Fail code       500               Sample JSON response              {  "status": "OK",  "message": "",  "body": {    "zeppelin.clock": [      {        "registry": "local",        "pkg": {          "type": "APPLICATION",          "name": "zeppelin.clock",          "description": "Clock (example)",          "artifact": "zeppelin-examples/zeppelin-example-clock/target/zeppelin-example-clock-0.7.0-SNAPSHOT.jar",          "className": "org.apache.zeppelin.example.app.clock.Clock",          "resources": [            [              ":java.util.Date"            ]          ],          "icon": "icon"        },        "enabled": false      }    ],    "zeppelin-bubblechart": [      {        "registry": "local",        "pkg": {          "type": "VISUALIZATION",          "name": "zeppelin-bubblechart",          "description": "Animated bubble chart",          "artifact": "./../helium/zeppelin-bubble",          "icon": "icon"        },        "enabled": true      },      {        "registry": "local",        "pkg": {          "type": "VISUALIZATION",          "name": "zeppelin-bubblechart",          "description": "Animated bubble chart",          "artifact": "zeppelin-bubblechart@0.0.2",          "icon": "icon"        },        "enabled": false      }    ],    "zeppelin_horizontalbar": [      {        "registry": "local",        "pkg": {          "type": "VISUALIZATION",          "name": "zeppelin_horizontalbar",          "description": "Horizontal Bar chart (example)",          "artifact": "./zeppelin-examples/zeppelin-example-horizontalbar",          "icon": "icon"        },        "enabled": true      }    ]  }}                    Suggest Helium application              Description      This GET method returns suggested helium application for the paragraph.              URL      http://[zeppelin-server]:[zeppelin-port]/api/helium/suggest/[Note ID]/[Paragraph ID]              Success code      200              Fail code              404 on note or paragraph not exists         500                    Sample JSON response              {  "status": "OK",  "message": "",  "body": {    "available": [      {        "registry": "local",        "pkg": {          "type": "APPLICATION",          "name": "zeppelin.clock",          "description": "Clock (example)",          "artifact": "zeppelin-examples/zeppelin-example-clock/target/zeppelin-example-clock-0.7.0-SNAPSHOT.jar",          "className": "org.apache.zeppelin.example.app.clock.Clock",          "resources": [            [              ":java.util.Date"            ]          ],          "icon": "icon"        },        "enabled": true      }    ]  }}                    Load helium Application on a paragraph              Description      This GET method returns a helium Application id on success.              URL      http://[zeppelin-server]:[zeppelin-port]/api/helium/load/[Note ID]/[Paragraph ID]              Success code      200              Fail code                404 on note or paragraph not exists           500 for any other errors                    Sample JSON response              {  "status": "OK",  "message": "",  "body": "app2C5FYRZ1E-20170108-0404492068241472zeppelin_clock"}                    Load bundled visualization script              Description      This GET method returns bundled helium visualization javascript. When refresh=true (optional) is provided, Zeppelin rebuild bundle. otherwise, provided from cache              URL      http://[zeppelin-server]:[zeppelin-port]/api/helium/visualizations/load[?refresh=true]              Success code      200 reponse body is executable javascript              Fail code                200 reponse body is error message string starts with ERROR:            Enable package              Description      This POST method enables a helium package. Needs artifact name in input payload              URL      http://[zeppelin-server]:[zeppelin-port]/api/helium/enable/[Package Name]              Success code      200              Fail code       500               Sample input              zeppelin-examples/zeppelin-example-clock/target/zeppelin-example-clock-0.7.0-SNAPSHOT.jar                            Sample JSON response              {"status":"OK"}                    Disable package              Description      This POST method disables a helium package.              URL      http://[zeppelin-server]:[zeppelin-port]/api/helium/disable/[Package Name]              Success code      200               Fail code       500               Sample JSON response              {"status":"OK"}            Get visualization display order              Description      This GET method returns display order of enabled visualization packages.              URL      http://[zeppelin-server]:[zeppelin-port]/api/helium/visualizationOrder              Success code      200               Fail code       500               Sample JSON response              {"status":"OK","body":["zeppelin_horizontalbar","zeppelin-bubblechart"]}            Set visualization display order              Description      This POST method sets visualization packages display order.              URL      http://[zeppelin-server]:[zeppelin-port]/api/helium/visualizationOrder              Success code      200               Fail code       500               Sample JSON input              ["zeppelin-bubblechart", "zeppelin_horizontalbar"]                    Sample JSON response              {"status":"OK"}            ",
      "url": " /rest-api/rest-helium.html",
      "group": "rest-api",
      "excerpt": "This page contains Apache Zeppelin Helium REST API information."
    }
    ,
    
  

    "/rest-api/rest-interpreter.html": {
      "title": "Apache Zeppelin Interpreter REST API",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Zeppelin Interpreter REST APIOverviewApache Zeppelin provides several REST APIs for interaction and remote activation of zeppelin functionality.All REST APIs are available starting with the following endpoint http://[zeppelin-server]:[zeppelin-port]/api. Note that Apache Zeppelin REST APIs receive or return JSON objects, it is recommended for you to install some JSON viewers such as JSONView.If you work with Apache Zeppelin and find a need for an additional REST API, please file an issue or send us an email.Interpreter REST API ListThe role of registered interpreters, settings and interpreters group are described in here.List of registered interpreters              Description      This GET method returns all the registered interpreters available on the server.              URL      http://[zeppelin-server]:[zeppelin-port]/api/interpreter              Success code      200              Fail code       500               Sample JSON response              {  "status": "OK",  "message": "",  "body": {    "md.md": {      "name": "md",      "group": "md",      "className": "org.apache.zeppelin.markdown.Markdown",      "properties": {},      "path": "/zeppelin/interpreter/md"    },    "spark.spark": {      "name": "spark",      "group": "spark",      "className": "org.apache.zeppelin.spark.SparkInterpreter",      "properties": {        "spark.executor.memory": {          "defaultValue": "1g",          "description": "Executor memory per worker instance. ex) 512m, 32g"        },        "spark.cores.max": {          "defaultValue": "",          "description": "Total number of cores to use. Empty value uses all available core."        },      },      "path": "/zeppelin/interpreter/spark"    },    "spark.sql": {      "name": "sql",      "group": "spark",      "className": "org.apache.zeppelin.spark.SparkSqlInterpreter",      "properties": {        "zeppelin.spark.maxResult": {          "defaultValue": "1000",          "description": "Max number of Spark SQL result to display."        }      },      "path": "/zeppelin/interpreter/spark"    }  }}                    List of registered interpreter settings              Description      This GET method returns all the interpreters settings registered on the server.              URL      http://[zeppelin-server]:[zeppelin-port]/api/interpreter/setting              Success code      200              Fail code       500               Sample JSON response              {  "status": "OK",  "message": "",  "body": [    {      "id": "2AYUGP2D5",      "name": "md",      "group": "md",      "properties": {        "empty": ""      },      "interpreterGroup": [        {          "class": "org.apache.zeppelin.markdown.Markdown",          "name": "md"        }      ],      "dependencies": []    },    {      "id": "2AY6GV7Q3",      "name": "spark",      "group": "spark",      "properties": {        "spark.cores.max": "",        "spark.executor.memory": "1g",      },      "interpreterGroup": [        {          "class": "org.apache.zeppelin.spark.SparkInterpreter",          "name": "spark"        },        {          "class": "org.apache.zeppelin.spark.SparkSqlInterpreter",          "name": "sql"        }      ],      "dependencies": [        {          "groupArtifactVersion": "com.databricks:spark-csv_2.10:1.3.0"        }      ]    }  ]}                    Get a registered interpreter setting by the setting id              Description      This GET method returns a registered interpreter setting on the server.              URL      http://[zeppelin-server]:[zeppelin-port]/api/interpreter/setting/[setting ID]              Success code      200              Fail code                400 if such interpreter setting id does not exist           500 for any other errors                    Sample JSON response              {  "status": "OK",  "message": "",  "body": {    "id": "2AYW25ANY",    "name": "Markdown setting name",    "group": "md",    "properties": {      "propname": "propvalue"    },    "interpreterGroup": [      {        "class": "org.apache.zeppelin.markdown.Markdown",        "name": "md"      }    ],    "dependencies": [      {        "groupArtifactVersion": "groupId:artifactId:version",        "exclusions": [          "groupId:artifactId"        ]      }    ]  }}                    Create a new interpreter setting              Description      This POST method adds a new interpreter setting using a registered interpreter to the server.              URL      http://[zeppelin-server]:[zeppelin-port]/api/interpreter/setting              Success code      201              Fail code                400 if the input json is empty           500 for any other errors                    Sample JSON input              {  "name": "Markdown setting name",  "group": "md",  "properties": {    "propname": "propvalue"  },  "interpreterGroup": [    {      "class": "org.apache.zeppelin.markdown.Markdown",      "name": "md"    }  ],  "dependencies": [    {      "groupArtifactVersion": "groupId:artifactId:version",      "exclusions": [        "groupId:artifactId"      ]    }  ]}                            Sample JSON response              {  "status": "CREATED",  "message": "",  "body": {    "id": "2AYW25ANY",    "name": "Markdown setting name",    "group": "md",    "properties": {      "propname": "propvalue"    },    "interpreterGroup": [      {        "class": "org.apache.zeppelin.markdown.Markdown",        "name": "md"      }    ],    "dependencies": [      {        "groupArtifactVersion": "groupId:artifactId:version",        "exclusions": [          "groupId:artifactId"        ]      }    ]  }}                    Update an interpreter setting              Description      This PUT method updates an interpreter setting with new properties.              URL      http://[zeppelin-server]:[zeppelin-port]/api/interpreter/setting/[interpreter ID]              Success code      200              Fail code       500               Sample JSON input              {  "name": "Markdown setting name",  "group": "md",  "properties": {    "propname": "Otherpropvalue"  },  "interpreterGroup": [    {      "class": "org.apache.zeppelin.markdown.Markdown",      "name": "md"    }  ],  "dependencies": [    {      "groupArtifactVersion": "groupId:artifactId:version",      "exclusions": [        "groupId:artifactId"      ]    }  ]}                            Sample JSON response              {  "status": "OK",  "message": "",  "body": {    "id": "2AYW25ANY",    "name": "Markdown setting name",    "group": "md",    "properties": {      "propname": "Otherpropvalue"    },    "interpreterGroup": [      {        "class": "org.apache.zeppelin.markdown.Markdown",        "name": "md"      }    ],    "dependencies": [      {        "groupArtifactVersion": "groupId:artifactId:version",        "exclusions": [          "groupId:artifactId"        ]      }    ]  }}                    Delete an interpreter setting              Description      This DELETE method deletes an given interpreter setting.              URL      http://[zeppelin-server]:[zeppelin-port]/api/interpreter/setting/[interpreter ID]              Success code      200               Fail code       500               Sample JSON response              {"status":"OK"}            Restart an interpreter              Description      This PUT method restarts the given interpreter id.              URL      http://[zeppelin-server]:[zeppelin-port]/api/interpreter/setting/restart/[interpreter ID]              Success code      200              Fail code       500               Sample JSON input (Optional)              {  "noteId": "2AVQJVC8N"}                            Sample JSON response              {"status":"OK"}            Add a new repository for dependency resolving              Description      This POST method adds new repository.              URL      http://[zeppelin-server]:[zeppelin-port]/api/interpreter/repository              Success code      201              Fail code       500               Sample JSON input              {  "id": "securecentral",  "url": "https://repo1.maven.org/maven2",  "snapshot": false}                            Sample JSON response              {"status":"OK"}            Delete a repository for dependency resolving              Description      This DELETE method delete repository with given id.              URL      http://[zeppelin-server]:[zeppelin-port]/api/interpreter/repository/[repository ID]              Success code      200              Fail code       500       ",
      "url": " /rest-api/rest-interpreter.html",
      "group": "rest-api",
      "excerpt": "This page contains Apache Zeppelin Interpreter REST API information."
    }
    ,
    
  

    "/rest-api/rest-notebook.html": {
      "title": "Apache Zeppelin Notebook REST API",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Zeppelin Notebook REST APIOverviewApache Zeppelin provides several REST APIs for interaction and remote activation of zeppelin functionality.All REST APIs are available starting with the following endpoint http://[zeppelin-server]:[zeppelin-port]/api. Note that Apache Zeppelin REST APIs receive or return JSON objects, it is recommended for you to install some JSON viewers such as JSONView.If you work with Apache Zeppelin and find a need for an additional REST API, please file an issue or send us an email.Notebooks REST API supports the following operations: List, Create, Get, Delete, Clone, Run, Export, Import as detailed in the following tables.Note operationsList of the notes              Description      This GET method lists the available notes on your server.          Notebook JSON contains the name and id of all notes.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook              Success code      200               Fail code       500                sample JSON response       {  "status": "OK",  "message": "",  "body": [    {      "name":"Homepage",      "id":"2AV4WUEMK"    },    {      "name":"Zeppelin Tutorial",      "id":"2A94M5J1Z"    }  ]}      Create a new note              Description      This POST method creates a new note using the given name or default name if none given.          The body field of the returned JSON contains the new note id.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook              Success code      201               Fail code       500                sample JSON input (without paragraphs)       {"name": "name of new note"}               sample JSON input (with initial paragraphs)       {  "name": "name of new note",  "paragraphs": [    {      "title": "paragraph title1",      "text": "paragraph text1"    },    {      "title": "paragraph title2",      "text": "paragraph text2"    }  ]}               sample JSON response       {  "status": "CREATED",  "message": "",  "body": "2AZPHY918"}      Get the status of all paragraphs              Description      This GET method gets the status of all paragraphs by the given note id.          The body field of the returned JSON contains of the array that compose of the paragraph id, paragraph status, paragraph finish date, paragraph started date.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/job/[noteId]              Success code      200               Fail code       500                sample JSON response       {  "status": "OK",  "body": [    {      "id":"20151121-212654_766735423",      "status":"FINISHED",      "finished":"Tue Nov 24 14:21:40 KST 2015",      "started":"Tue Nov 24 14:21:39 KST 2015"    },    {      "progress":"1",      "id":"20151121-212657_730976687",      "status":"RUNNING",      "finished":"Tue Nov 24 14:21:35 KST 2015",      "started":"Tue Nov 24 14:21:40 KST 2015"    }  ]}      Get an existing note information              Description      This GET method retrieves an existing note's information using the given id.          The body field of the returned JSON contain information about paragraphs in the note.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]              Success code      200               Fail code       500                sample JSON response       {  "status": "OK",  "message": "",  "body": {    "paragraphs": [      {        "text": "%sql nselect age, count(1) valuenfrom bank nwhere age < 30 ngroup by age norder by age",        "config": {          "colWidth": 4,          "graph": {            "mode": "multiBarChart",            "height": 300,            "optionOpen": false,            "keys": [              {                "name": "age",                "index": 0,                "aggr": "sum"              }            ],            "values": [              {                "name": "value",                "index": 1,                "aggr": "sum"              }            ],            "groups": [],            "scatter": {              "xAxis": {                "name": "age",                "index": 0,                "aggr": "sum"              },              "yAxis": {                "name": "value",                "index": 1,                "aggr": "sum"              }            }          }        },        "settings": {          "params": {},          "forms": {}        },        "jobName": "paragraph_1423500782552_-1439281894",        "id": "20150210-015302_1492795503",        "results": {          "code": "SUCCESS",          "msg": [            {              "type": "TABLE",              "data": "agetvaluen19t4n20t3n21t7n22t9n23t20n24t24n25t44n26t77n27t94n28t103n29t97n"            }          ]        },        "dateCreated": "Feb 10, 2015 1:53:02 AM",        "dateStarted": "Jul 3, 2015 1:43:17 PM",        "dateFinished": "Jul 3, 2015 1:43:23 PM",        "status": "FINISHED",        "progressUpdateIntervalMs": 500      }    ],    "name": "Zeppelin Tutorial",    "id": "2A94M5J1Z",    "angularObjects": {},    "config": {      "looknfeel": "default"    },    "info": {}  }}      Delete a note              Description      This DELETE method deletes a note by the given note id.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]              Success code      200               Fail code       500                sample JSON response       {"status": "OK","message": ""}      Clone a note              Description      This POST method clones a note by the given id and create a new note using the given name          or default name if none given.          The body field of the returned JSON contains the new note id.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]              Success code      201               Fail code       500                sample JSON input       {"name": "name of new note"}               sample JSON response       {  "status": "CREATED",  "message": "",  "body": "2AZPHY918"}      Export a note              Description      This GET method exports a note by the given id and gernerates a JSON                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/export/[noteId]              Success code      201               Fail code       500          sample JSON response       {  "paragraphs": [    {      "text": "%md This is my new paragraph in my new note",      "dateUpdated": "Jan 8, 2016 4:49:38 PM",      "config": {        "enabled": true      },      "settings": {        "params": {},        "forms": {}      },      "jobName": "paragraph_1452300578795_1196072540",      "id": "20160108-164938_1685162144",      "dateCreated": "Jan 8, 2016 4:49:38 PM",      "status": "READY",      "progressUpdateIntervalMs": 500    }  ],  "name": "source note for export",  "id": "2B82H3RR1",  "angularObjects": {},  "config": {},  "info": {}}      Import a note              Description      This POST method imports a note from the note JSON input                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/import              Success code      201               Fail code       500               sample JSON input      {  "paragraphs": [    {      "text": "%md This is my new paragraph in my new note",      "dateUpdated": "Jan 8, 2016 4:49:38 PM",      "config": {        "enabled": true      },      "settings": {        "params": {},        "forms": {}      },      "jobName": "paragraph_1452300578795_1196072540",      "id": "20160108-164938_1685162144",      "dateCreated": "Jan 8, 2016 4:49:38 PM",      "status": "READY",      "progressUpdateIntervalMs": 500    }  ],  "name": "source note for export",  "id": "2B82H3RR1",  "angularObjects": {},  "config": {},  "info": {}}              sample JSON response      {  "status": "CREATED",  "message": "",  "body": "2AZPHY918"}      Run all paragraphs              Description            This POST method runs all paragraphs in the given note id.       If you can not find Note id 404 returns.      If there is a problem with the interpreter returns a 412 error.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/job/[noteId]              Success code      200               Fail code       404 or 412               sample JSON response       {"status": "OK"}                sample JSON error response                            {             "status": "NOTFOUND",             "message": "note not found."           }                             {             "status": "PRECONDITIONFAILED",             "message": "paragraph1469771130099-278315611 Not selected or Invalid Interpreter bind"           }                      Stop all paragraphs              Description      This DELETE method stops all paragraphs in the given note id.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/job/[noteId]              Success code      200               Fail code       500                sample JSON response       {"status":"OK"}      Clear all paragraph result              Description      This PUT method clear all paragraph results from note of given id.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/clear              Success code      200              Forbidden code      401              Not Found code      404              Fail code      500              sample JSON response      {"status": "OK"}          Paragraph operationsCreate a new paragraph              Description      This POST method create a new paragraph using JSON payload.          The body field of the returned JSON contain the new paragraph id.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/paragraph              Success code      201               Fail code       500                sample JSON input (add to the last)       {  "title": "Paragraph insert revised",  "text": "%sparknprintln("Paragraph insert revised")"}               sample JSON input (add to specific index)       {  "title": "Paragraph insert revised",  "text": "%sparknprintln("Paragraph insert revised")",  "index": 0}               sample JSON response       {  "status": "CREATED",  "message": "",  "body": "20151218-100330_1754029574"}      Get a paragraph information              Description      This GET method retrieves an existing paragraph's information using the given id.          The body field of the returned JSON contain information about paragraph.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/paragraph/[paragraphId]              Success code      200               Fail code       500                sample JSON response       {  "status": "OK",  "message": "",  "body": {    "title": "Paragraph2",    "text": "%sparknnprintln("it's paragraph2")",    "dateUpdated": "Dec 18, 2015 7:33:54 AM",    "config": {      "colWidth": 12,      "graph": {        "mode": "table",        "height": 300,        "optionOpen": false,        "keys": [],        "values": [],        "groups": [],        "scatter": {}      },      "enabled": true,      "title": true,      "editorMode": "ace/mode/scala"    },    "settings": {      "params": {},      "forms": {}    },    "jobName": "paragraph_1450391574392_-1890856722",    "id": "20151218-073254_1105602047",    "results": {      "code": "SUCCESS",      "msg": [        {           "type": "TEXT",           "data": "it's paragraph2n"        }      ]    },    "dateCreated": "Dec 18, 2015 7:32:54 AM",    "dateStarted": "Dec 18, 2015 7:33:55 AM",    "dateFinished": "Dec 18, 2015 7:33:55 AM",    "status": "FINISHED",    "progressUpdateIntervalMs": 500  }}      Get the status of a single paragraph              Description      This GET method gets the status of a single paragraph by the given note and paragraph id.          The body field of the returned JSON contains of the array that compose of the paragraph id, paragraph status, paragraph finish date, paragraph started date.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/job/[noteId]/[paragraphId]              Success code      200               Fail code       500                sample JSON response       {  "status": "OK",  "body": {      "id":"20151121-212654_766735423",      "status":"FINISHED",      "finished":"Tue Nov 24 14:21:40 KST 2015",      "started":"Tue Nov 24 14:21:39 KST 2015"    }}      Update paragraph configuration              Description      This PUT method update paragraph configuration using given id so that user can change paragraph setting such as graph type, show or hide editor/result and paragraph size, etc. You can update certain fields you want, for example you can update colWidth field only by sending request with payload {"colWidth": 12.0}.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/paragraph/[paragraphId]/config              Success code      200              Bad Request code      400              Forbidden code      403              Not Found code      404              Fail code      500              sample JSON input      {  "colWidth": 6.0,  "graph": {    "mode": "lineChart",    "height": 200.0,    "optionOpen": false,    "keys": [      {        "name": "age",        "index": 0.0,        "aggr": "sum"      }    ],    "values": [      {        "name": "value",        "index": 1.0,        "aggr": "sum"      }    ],    "groups": [],    "scatter": {}  },  "editorHide": true,  "editorMode": "ace/mode/markdown",  "tableHide": false}              sample JSON response      {  "status":"OK",  "message":"",  "body":{    "text":"%sql nselect age, count(1) valuenfrom bank nwhere age u003c 30 ngroup by age norder by age",    "config":{      "colWidth":6.0,      "graph":{        "mode":"lineChart",        "height":200.0,        "optionOpen":false,        "keys":[          {            "name":"age",            "index":0.0,            "aggr":"sum"          }        ],        "values":[          {            "name":"value",            "index":1.0,            "aggr":"sum"          }        ],        "groups":[],        "scatter":{}      },      "tableHide":false,      "editorMode":"ace/mode/markdown",      "editorHide":true    },    "settings":{      "params":{},      "forms":{}    },    "apps":[],    "jobName":"paragraph1423500782552-1439281894",    "id":"20150210-015302_1492795503",    "results":{      "code":"SUCCESS",      "msg": [        {          "type":"TABLE",          "data":"agetvaluen19t4n20t3n21t7n22t9n23t20n24t24n25t44n26t77n27t94n28t103n29t97n"        }      ]    },    "dateCreated":"Feb 10, 2015 1:53:02 AM",    "dateStarted":"Jul 3, 2015 1:43:17 PM",    "dateFinished":"Jul 3, 2015 1:43:23 PM",    "status":"FINISHED",    "progressUpdateIntervalMs":500  }}      Delete a paragraph              Description      This DELETE method deletes a paragraph by the given note and paragraph id.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/paragraph/[paragraphId]              Success code      200               Fail code       500                sample JSON response       {"status": "OK","message": ""}      Run a paragraph asynchronously              Description      This POST method runs the paragraph asynchronously by given note and paragraph id. This API always return SUCCESS even if the execution of the paragraph fails later because the API is asynchronous                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/job/[noteId]/[paragraphId]              Success code      200               Fail code       500                sample JSON input (optional, only needed when if you want to update dynamic form's value)       {  "name": "name of new note",  "params": {    "formLabel1": "value1",    "formLabel2": "value2"  }}               sample JSON response       {"status": "OK"}      Run a paragraph synchronously              Description      This POST method runs the paragraph synchronously by given note and paragraph id. This API can return SUCCESS or ERROR depending on the outcome of the paragraph execution                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/run/[noteId]/[paragraphId]              Success code      200               Fail code       500                sample JSON input (optional, only needed when if you want to update dynamic form's value)       {  "name": "name of new note",  "params": {    "formLabel1": "value1",    "formLabel2": "value2"  }}               sample JSON response       {"status": "OK"}               sample JSON error       {   "status": "INTERNAL_SERVER_ERROR",   "body": {       "code": "ERROR",       "type": "TEXT",       "msg": "bash: -c: line 0: unexpected EOF while looking for matching ``'nbash: -c: line 1: syntax error: unexpected end of filenExitValue: 2"   }}      Stop a paragraph              Description      This DELETE method stops the paragraph by given note and paragraph id.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/job/[noteId]/[paragraphId]              Success code      200               Fail code       500                sample JSON response       {"status": "OK"}      Move a paragraph to the specific index              Description      This POST method moves a paragraph to the specific index (order) from the note.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/paragraph/[paragraphId]/move/[newIndex]              Success code      200               Fail code       500                sample JSON response       {"status": "OK","message": ""}      Full text search through the paragraphs in all notes              Description      GET request will return list of matching paragraphs                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/search?q=[query]              Success code      200              Fail code       500               Sample JSON response       {  "status": "OK",  "body": [    {      "id": "/paragraph/",      "name":"Note Name",       "snippet":"",      "text":""    }  ]}      Cron jobsAdd Cron Job              Description      This POST method adds cron job by the given note id.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/cron/[noteId]              Success code      200               Fail code       500                sample JSON input       {"cron": "cron expression of note"}               sample JSON response       {"status": "OK"}      Remove Cron Job              Description      This DELETE method removes cron job by the given note id.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/cron/[noteId]              Success code      200               Fail code       500                sample JSON response       {"status": "OK"}      Get Cron Job              Description      This GET method gets cron job expression of given note id.          The body field of the returned JSON contains the cron expression.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/cron/[noteId]              Success code      200               Fail code       500                sample JSON response       {"status": "OK", "body": "* * * * * ?"}      PermissionGet a note permission information              Description      This GET method gets a note authorization information.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/permissions              Success code      200              Forbidden code      403              Fail code      500               sample JSON response       {   "status":"OK",   "message":"",   "body":{      "readers":[         "user2"      ],      "owners":[         "user1"      ],      "writers":[         "user2"      ]   }}            Set note permission              Description      This PUT method set note authorization information.                    URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook/[noteId]/permissions              Success code      200              Forbidden code      403              Fail code      500               sample JSON input       {  "readers": [    "user1"  ],  "owners": [    "user2"  ],  "writers": [    "user1"  ]}               sample JSON response       {  "status": "OK"}      ",
      "url": " /rest-api/rest-notebook.html",
      "group": "rest-api",
      "excerpt": "This page contains Apache Zeppelin Notebook REST API information."
    }
    ,
    
  

    "/rest-api/rest-notebookRepo.html": {
      "title": "Apache Zeppelin notebook repository REST API",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->Apache Zeppelin Notebook Repository APIOverviewApache Zeppelin provides several REST APIs for interaction and remote activation of zeppelin functionality.All REST APIs are available starting with the following endpoint http://[zeppelin-server]:[zeppelin-port]/api. Note that Apache Zeppelin REST APIs receive or return JSON objects, it is recommended for you to install some JSON viewers such as JSONView.If you work with Apache Zeppelin and find a need for an additional REST API, please file an issue or send us an email.Notebook Repository REST API ListList all available notebook repositories              Description      This GET method returns all the available notebook repositories.              URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook-repositories              Success code      200              Fail code      500              Sample JSON response              {  "status": "OK",  "message": "",  "body": [    {      "name": "GitNotebookRepo",      "className": "org.apache.zeppelin.notebook.repo.GitNotebookRepo",      "settings": [        {          "type": "INPUT",          "value": [],          "selected": "ZEPPELIN_HOME/zeppelin/notebook/",          "name": "Notebook Path"        }      ]    }  ]}                    Reload a notebook repository              Description      This GET method triggers reloading and broadcasting of the note list.              URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook-repositories/reload              Success code      200              Fail code      500              Sample JSON response              {  "status": "OK",  "message": ""}                    Update a specific notebook repository              Description      This PUT method updates a specific notebook repository.              URL      http://[zeppelin-server]:[zeppelin-port]/api/notebook-repositories              Success code      200              Fail code              404 when the specified notebook repository doesn't exist          406 for invalid payload         500 for any other errors                    Sample JSON input              {  "name":"org.apache.zeppelin.notebook.repo.GitNotebookRepo",  "settings":{    "Notebook Path":"/tmp/notebook/"  }}                            Sample JSON response              {  "status": "OK",  "message": "",  "body": {    "name": "GitNotebookRepo",    "className": "org.apache.zeppelin.notebook.repo.GitNotebookRepo",    "settings": [      {        "type": "INPUT",        "value": [],        "selected": "/tmp/notebook/",        "name": "Notebook Path"      }    ]  }}                    ",
      "url": " /rest-api/rest-notebookRepo.html",
      "group": "rest-api",
      "excerpt": "This page contains Apache Zeppelin notebook repository REST API information."
    }
    ,
    
  
  
  
  
  

    "/security/authentication.html": {
      "title": "Authentication for NGINX",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Authentication for NGINX[Build in authentication mechanism](./shiroauthentication.html) is recommended way for authentication. In case of you want authenticate using NGINX and [HTTP basic auth](https://en.wikipedia.org/wiki/Basic_access_authentication), please read this document.## HTTP Basic Authentication using NGINX> **Quote from Wikipedia:** NGINX is a web server. It can act as a reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer and an HTTP cache.So you can use NGINX server as proxy server to serve HTTP Basic Authentication as a separate process along with Zeppelin server.Here are instructions how to accomplish the setup NGINX as a front-end authentication server and connect Zeppelin at behind.This instruction based on Ubuntu 14.04 LTS but may work with other OS with few configuration changes.1. Install NGINX server on your server instance    You can install NGINX server with same box where zeppelin installed or separate box where it is dedicated to serve as proxy server.    ```    $ apt-get install nginx    ```    > **NOTE :** On pre 1.3.13 version of NGINX, Proxy for Websocket may not fully works. Please use latest version of NGINX. See: [NGINX documentation](https://www.nginx.com/blog/websocket-nginx/).1. Setup init script in NGINX    In most cases, NGINX configuration located under `/etc/nginx/sites-available`. Create your own configuration or add your existing configuration at `/etc/nginx/sites-available`.    ```    $ cd /etc/nginx/sites-available    $ touch my-zeppelin-auth-setting    ```    Now add this script into `my-zeppelin-auth-setting` file. You can comment out `optional` lines If you want serve Zeppelin under regular HTTP 80 Port.    ```    upstream zeppelin {        server [YOUR-ZEPPELIN-SERVER-IP]:[YOUR-ZEPPELIN-SERVER-PORT];   # For security, It is highly recommended to make this address/port as non-public accessible    }    # Zeppelin Website    server {        listen [YOUR-ZEPPELIN-WEB-SERVER-PORT];        listen 443 ssl;                                      # optional, to serve HTTPS connection        server_name [YOUR-ZEPPELIN-SERVER-HOST];             # for example: zeppelin.mycompany.com        ssl_certificate [PATH-TO-YOUR-CERT-FILE];            # optional, to serve HTTPS connection        ssl_certificate_key [PATH-TO-YOUR-CERT-KEY-FILE];    # optional, to serve HTTPS connection        if ($ssl_protocol = "") {            rewrite ^ https://$host$request_uri? permanent;  # optional, to force use of HTTPS        }        location / {    # For regular websever support            proxy_pass http://zeppelin;            proxy_set_header X-Real-IP $remote_addr;            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;            proxy_set_header Host $http_host;            proxy_set_header X-NginX-Proxy true;            proxy_redirect off;            auth_basic "Restricted";            auth_basic_user_file /etc/nginx/.htpasswd;        }        location /ws {  # For websocket support            proxy_pass http://zeppelin/ws;            proxy_http_version 1.1;            proxy_set_header Upgrade websocket;            proxy_set_header Connection upgrade;            proxy_read_timeout 86400;        }    }    ```    Then make a symbolic link to this file from `/etc/nginx/sites-enabled/` to enable configuration above when NGINX reloads.    ```    $ ln -s /etc/nginx/sites-enabled/my-zeppelin-auth-setting /etc/nginx/sites-available/my-zeppelin-auth-setting    ```1. Setup user credential into `.htpasswd` file and restart server    Now you need to setup `.htpasswd` file to serve list of authenticated user credentials for NGINX server.    ```    $ cd /etc/nginx    $ htpasswd -c htpasswd [YOUR-ID]    $ NEW passwd: [YOUR-PASSWORD]    $ RE-type new passwd: [YOUR-PASSWORD-AGAIN]    ```    Or you can use your own apache `.htpasswd` files in other location for setting up property: `auth_basic_user_file`    Restart NGINX server.    ```    $ service nginx restart    ```    Then check HTTP Basic Authentication works in browser. If you can see regular basic auth popup and then able to login with credential you entered into `.htpasswd` you are good to go.1. More security consideration* Using HTTPS connection with Basic Authentication is highly recommended since basic auth without encryption may expose your important credential information over the network.* Using [Shiro Security feature built-into Zeppelin](./shiroauthentication.html) is recommended if you prefer all-in-one solution for authentication but NGINX may provides ad-hoc solution for re-use authentication served by your system's NGINX server or in case of you need to separate authentication from zeppelin server.* It is recommended to isolate direct connection to Zeppelin server from public internet or external services to secure your zeppelin instance from unexpected attack or problems caused by public zone.## Another optionAnother option is to have an authentication server that can verify user credentials in an LDAP server.If an incoming request to the Zeppelin server does not have a cookie with user information encrypted with the authentication server public key, the useris redirected to the authentication server. Once the user is verified, the authentication server redirects the browser to a specific URL in the Zeppelin server which sets the authentication cookie in the browser.The end result is that all requests to the Zeppelin web server have the authentication cookie which contains user and groups information.",
      "url": " /security/authentication.html",
      "group": "security",
      "excerpt": "There are multiple ways to enable authentication in Apache Zeppelin. This page describes HTTP basic auth using NGINX."
    }
    ,
    
  

    "/security/datasource_authorization.html": {
      "title": "Data Source Authorization in Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Data Source Authorization in Apache Zeppelin## OverviewData source authorization involves authenticating to the data source like a Mysql database and letting it determine user permissions.Apache Zeppelin allows users to use their own credentials to authenticate with **Data Sources**.For example, let's assume you have an account in the Vertica databases with credentials. You might want to use this account to create a JDBC connection instead of a shared account with all users who are defined in `conf/shiro.ini`. In this case, you can add your credential information to Apache Zeppelin and use them with below simple steps.  ## How to save the credential information?You can add new credentials in the dropdown menu for your data source which can be passed to interpreters. **Entity** can be the key that distinguishes each credential sets.(We suggest that the convention of the **Entity** is `[Interpreter Group].[Interpreter Name]`.)Please see [what is interpreter group](../manual/interpreters.html#what-is-interpreter-group) for the detailed information.Type **Username & Password** for your own credentials. ex) Mysql user & password of the JDBC Interpreter.The credentials saved as per users defined in `conf/shiro.ini`.If you didn't activate [shiro authentication in Apache Zeppelin](./shiroauthentication.html), your credential information will be saved as `anonymous`.All credential information also can be found in `conf/credentials.json`. #### JDBC interpreterYou need to maintain per-user connection pools.The interpret method takes the user string as a parameter and executes the jdbc call using a connection in the user's connection pool.#### Presto You don't need a password if the Presto DB server runs backend code using HDFS authorization for the user.#### Vertica and Mysql You have to store the password information for users.## Please noteAs a first step of data source authentication feature, [ZEPPELIN-828](https://issues.apache.org/jira/browse/ZEPPELIN-828) was proposed and implemented in Pull Request [#860](https://github.com/apache/zeppelin/pull/860).Currently, only customized 3rd party interpreters can use this feature. We are planning to apply this mechanism to [the community managed interpreters](../manual/interpreterinstallation.html#available-community-managed-interpreters) in the near future. Please keep track [ZEPPELIN-1070](https://issues.apache.org/jira/browse/ZEPPELIN-1070). ",
      "url": " /security/datasource_authorization.html",
      "group": "security",
      "excerpt": "Apache Zeppelin supports protected data sources. In case of a MySql database, every users can set up their own credentials to access it."
    }
    ,
    
  

    "/security/helium_authorization.html": {
      "title": "Helium Authorization in Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Helium Authorization in Apache Zeppelin## How to configure proxies?Set **http_proxy** and **https_proxy** env variables to allow connection to npm registry behind a corporate firewall.",
      "url": " /security/helium_authorization.html",
      "group": "security",
      "excerpt": "Apache Zeppelin supports Helium plugins which fetch required installer packages from remote registry/repositories"
    }
    ,
    
  

    "/security/notebook_authorization.html": {
      "title": "Notebook Authorization in Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Zeppelin Notebook Authorization## OverviewWe assume that there is an **Shiro Authentication** component that associates a user string and a set of group strings with every NotebookSocket.If you don't set the authentication components yet, please check [Shiro authentication for Apache Zeppelin](./shiroauthentication.html) first.## Authorization SettingYou can set Zeppelin notebook permissions in each notebooks. Of course only **notebook owners** can change this configuration.Just click **Lock icon** and open the permission setting page in your notebook.As you can see, each Zeppelin notebooks has 3 entities :* Owners ( users or groups )* Readers ( users or groups )* Writers ( users or groups )Fill out the each forms with comma seperated **users** and **groups** configured in `conf/shiro.ini` file.If the form is empty (*), it means that any users can perform that operation.If someone who doesn't have **read** permission is trying to access the notebook or someone who doesn't have **write** permission is trying to edit the notebook, Zeppelin will ask to login or block the user.## Separate notebook workspaces (public vs. private)By default, the authorization rights allow other users to see the newly created note, meaning the workspace is `public`. This behavior is controllable and can be set through either `ZEPPELIN_NOTEBOOK_PUBLIC` variable in `conf/zeppelin-env.sh`, or through `zeppelin.notebook.public` property in `conf/zeppelin-site.xml`. Thus, in order to make newly created note appear only in your `private` workspace by default, you can set either `ZEPPELIN_NOTEBOOK_PUBLIC` to `false` in your `conf/zeppelin-env.sh` as follows:```export ZEPPELIN_NOTEBOOK_PUBLIC="false"```or set `zeppelin.notebook.public` property to `false` in `conf/zeppelin-site.xml` as follows:```  zeppelin.notebook.public  false  Make notebook public by default when created, private otherwise```Behind the scenes, when you create a new note only the `owners` field is filled with current user, leaving `readers` and `writers` fields empty. All the notes with at least one empty authorization field are considered to be in `public` workspace. Thus when setting `zeppelin.notebook.public` (or corresponding `ZEPPELIN_NOTEBOOK_PUBLIC`) to false, newly created notes have `readers` and `writers` fields filled with current user, making note appear as in `private` workspace.## How it worksIn this section, we will explain the detail about how the notebook authorization works in backend side.### NotebookServerThe [NotebookServer](https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java) classifies every notebook operations into three categories: **Read**, **Write**, **Manage**.Before executing a notebook operation, it checks if the user and the groups associated with the `NotebookSocket` have permissions.For example, before executing a **Read** operation, it checks if the user and the groups have at least one entity that belongs to the **Reader** entities.### Notebook REST API callZeppelin executes a [REST API call](https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java) for the notebook permission information.In the backend side, Zeppelin gets the user information for the connection and allows the operation if the users and groupsassociated with the current user have at least one entity that belongs to owner entities for the notebook.",
      "url": " /security/notebook_authorization.html",
      "group": "security",
      "excerpt": "This page will guide you how you can set the permission for Zeppelin notebooks. This document assumes that Apache Shiro authentication was set up."
    }
    ,
    
  

    "/security/shiroauthentication.html": {
      "title": "Apache Shiro Authentication for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Apache Shiro authentication for Apache Zeppelin## Overview[Apache Shiro](http://shiro.apache.org/) is a powerful and easy-to-use Java security framework that performs authentication, authorization, cryptography, and session management. In this documentation, we will explain step by step how Shiro works for Zeppelin notebook authentication.When you connect to Apache Zeppelin, you will be asked to enter your credentials. Once you logged in, then you have access to all notes including other user's notes.## Security SetupYou can setup **Zeppelin notebook authentication** in some simple steps.### 1. Enable ShiroBy default in `conf`, you will find `shiro.ini.template`, this file is used as an example and it is strongly recommendedto create a `shiro.ini` file by doing the following command line```bashcp conf/shiro.ini.template conf/shiro.ini```For the further information about  `shiro.ini` file format, please refer to [Shiro Configuration](http://shiro.apache.org/configuration.html#Configuration-INISections).### 2. Secure the Websocket channelSet to property **zeppelin.anonymous.allowed** to **false** in `conf/zeppelin-site.xml`. If you don't have this file yet, just copy `conf/zeppelin-site.xml.template` to `conf/zeppelin-site.xml`.### 3. Start Zeppelin```bin/zeppelin-daemon.sh start (or restart)```Then you can browse Zeppelin at [http://localhost:8080](http://localhost:8080).### 4. LoginFinally, you can login using one of the below **username/password** combinations.```[users]admin = password1, adminuser1 = password2, role1, role2user2 = password3, role3user3 = password4, role2```You can set the roles for each users next to the password.## Groups and permissions (optional)In case you want to leverage user groups and permissions, use one of the following configuration for LDAP or AD under `[main]` segment in `shiro.ini`.```activeDirectoryRealm = org.apache.zeppelin.realm.ActiveDirectoryGroupRealmactiveDirectoryRealm.systemUsername = userNameAactiveDirectoryRealm.systemPassword = passwordAactiveDirectoryRealm.searchBase = CN=Users,DC=SOME_GROUP,DC=COMPANY,DC=COMactiveDirectoryRealm.url = ldap://ldap.test.com:389activeDirectoryRealm.groupRolesMap = "CN=aGroupName,OU=groups,DC=SOME_GROUP,DC=COMPANY,DC=COM":"group1"activeDirectoryRealm.authorizationCachingEnabled = falseactiveDirectoryRealm.principalSuffix = @corp.company.netldapRealm = org.apache.zeppelin.server.LdapGroupRealm# search base for ldap groups (only relevant for LdapGroupRealm):ldapRealm.contextFactory.environment[ldap.searchBase] = dc=COMPANY,dc=COMldapRealm.contextFactory.url = ldap://ldap.test.com:389ldapRealm.userDnTemplate = uid={0},ou=Users,dc=COMPANY,dc=COMldapRealm.contextFactory.authenticationMechanism = simple```also define roles/groups that you want to have in system, like below;```[roles]admin = *hr = *finance = *group1 = *```## Configure Realm (optional)Realms are responsible for authentication and authorization in Apache Zeppelin. By default, Apache Zeppelin uses [IniRealm](https://shiro.apache.org/static/latest/apidocs/org/apache/shiro/realm/text/IniRealm.html) (users and groups are configurable in `conf/shiro.ini` file under `[user]` and `[group]` section). You can also leverage Shiro Realms like [JndiLdapRealm](https://shiro.apache.org/static/latest/apidocs/org/apache/shiro/realm/ldap/JndiLdapRealm.html), [JdbcRealm](https://shiro.apache.org/static/latest/apidocs/org/apache/shiro/realm/jdbc/JdbcRealm.html) or create [our own](https://shiro.apache.org/static/latest/apidocs/org/apache/shiro/realm/AuthorizingRealm.html).To learn more about Apache Shiro Realm, please check [this documentation](http://shiro.apache.org/realm.html).We also provide community custom Realms.### Active Directory```activeDirectoryRealm = org.apache.zeppelin.realm.ActiveDirectoryGroupRealmactiveDirectoryRealm.systemUsername = userNameAactiveDirectoryRealm.systemPassword = passwordAactiveDirectoryRealm.hadoopSecurityCredentialPath = jceks://file/user/zeppelin/conf/zeppelin.jceksactiveDirectoryRealm.searchBase = CN=Users,DC=SOME_GROUP,DC=COMPANY,DC=COMactiveDirectoryRealm.url = ldap://ldap.test.com:389activeDirectoryRealm.groupRolesMap = "CN=aGroupName,OU=groups,DC=SOME_GROUP,DC=COMPANY,DC=COM":"group1"activeDirectoryRealm.authorizationCachingEnabled = falseactiveDirectoryRealm.principalSuffix = @corp.company.net```Also instead of specifying systemPassword in clear text in shiro.ini administrator can choose to specify the same in "hadoop credential".Create a keystore file using the hadoop credential commandline, for this the hadoop commons should be in the classpath`hadoop credential create activeDirectoryRealm.systempassword -provider jceks://file/user/zeppelin/conf/zeppelin.jceks`Change the following values in the Shiro.ini file, and uncomment the line:`activeDirectoryRealm.hadoopSecurityCredentialPath = jceks://file/user/zeppelin/conf/zeppelin.jceks`### LDAPTwo options exist for configuring an LDAP Realm. The simpler to use is the LdapGroupRealm. How ever it has limited flexibility with mapping of ldap groups to users and for authorization for user groups. A sample configuration file forthis realm is given below.```ldapRealm = org.apache.zeppelin.realm.LdapGroupRealm# search base for ldap groups (only relevant for LdapGroupRealm):ldapRealm.contextFactory.environment[ldap.searchBase] = dc=COMPANY,dc=COMldapRealm.contextFactory.url = ldap://ldap.test.com:389ldapRealm.userDnTemplate = uid={0},ou=Users,dc=COMPANY,dc=COMldapRealm.contextFactory.authenticationMechanism = simple```The other more flexible option is to use the LdapRealm. It allows for mapping of ldapgroups to roles and also allows for role/group based authentication into the zeppelin server. Sample configuration for this realm is given below. ```[main] ldapRealm=org.apache.zeppelin.realm.LdapRealmldapRealm.contextFactory.authenticationMechanism=simpleldapRealm.contextFactory.url=ldap://localhost:33389ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org# Ability to set ldap paging Size if needed default is 100ldapRealm.pagingSize = 200ldapRealm.authorizationEnabled=trueldapRealm.contextFactory.systemAuthenticationMechanism=simpleldapRealm.searchBase=dc=hadoop,dc=apache,dc=orgldapRealm.userSearchBase = dc=hadoop,dc=apache,dc=orgldapRealm.groupSearchBase = ou=groups,dc=hadoop,dc=apache,dc=orgldapRealm.groupObjectClass=groupofnames# Allow userSearchAttribute to be customizedldapRealm.userSearchAttributeName = sAMAccountNameldapRealm.memberAttribute=member# force usernames returned from ldap to lowercase useful for ADldapRealm.userLowerCase = true # ability set searchScopes subtree (default), one, baseldapRealm.userSearchScope = subtree;ldapRealm.groupSearchScope = subtree;ldapRealm.memberAttributeValueTemplate=cn={0},ou=people,dc=hadoop,dc=apache,dc=orgldapRealm.contextFactory.systemUsername=uid=guest,ou=people,dc=hadoop,dc=apache,dc=org ldapRealm.contextFactory.systemPassword=S{ALIAS=ldcSystemPassword}# enable support for nested groups using the LDAP_MATCHING_RULE_IN_CHAIN operatorldapRealm.groupSearchEnableMatchingRuleInChain = true# optional mapping from physical groups to logical application rolesldapRealm.rolesByGroup = LDN_USERS: user_role, NYK_USERS: user_role, HKG_USERS: user_role, GLOBAL_ADMIN: admin_role# optional list of roles that are allowed to authenticate. Incase not present all groups are allowed to authenticate (login). # This changes nothing for url specific permissions that will continue to work as specified in [urls].ldapRealm.allowedRolesForAuthentication = admin_role,user_roleldapRealm.permissionsByRole= user_role = *:ToDoItemsJdo:*:*, *:ToDoItem:*:*; admin_role = *securityManager.sessionManager = $sessionManagersecurityManager.realms = $ldapRealm ```### PAM[PAM](https://en.wikipedia.org/wiki/Pluggable_authentication_module) authentication support allows the reuse of existing authentication moduls on the host where Zeppelin is running. On a typical system modules are configured per service for example sshd, passwd, etc. under `/etc/pam.d/`. You caneither reuse one of these services or create your own for Zeppelin. Activiting PAM authentication requires two parameters: 1. realm: The Shiro realm being used 2. service: The service configured under `/etc/pam.d/` to be used. The name here needs to be the same as the file name under `/etc/pam.d/` ```[main] pamRealm=org.apache.zeppelin.realm.PamRealm pamRealm.service=sshd```### ZeppelinHub[ZeppelinHub](https://www.zeppelinhub.com) is a service that synchronize your Apache Zeppelin notebooks and enables you to collaborate easily.To enable login with your ZeppelinHub credential, apply the following change in `conf/shiro.ini` under `[main]` section.```### A sample for configuring ZeppelinHub RealmzeppelinHubRealm = org.apache.zeppelin.realm.ZeppelinHubRealm## Url of ZeppelinHubzeppelinHubRealm.zeppelinhubUrl = https://www.zeppelinhub.comsecurityManager.realms = $zeppelinHubRealm```> Note: ZeppelinHub is not releated to Apache Zeppelin project.## Secure your Zeppelin information (optional)By default, anyone who defined in `[users]` can share **Interpreter Setting**, **Credential** and **Configuration** information in Apache Zeppelin.Sometimes you might want to hide these information for your use case.Since Shiro provides **url-based security**, you can hide the information by commenting or uncommenting these below lines in `conf/shiro.ini`.```[urls]/api/interpreter/** = authc, roles[admin]/api/configurations/** = authc, roles[admin]/api/credential/** = authc, roles[admin]```In this case, only who have `admin` role can see **Interpreter Setting**, **Credential** and **Configuration** information.If you want to grant this permission to other users, you can change **roles[ ]** as you defined at `[users]` section.> **NOTE :** All of the above configurations are defined in the `conf/shiro.ini` file.## Other authentication methods- [HTTP Basic Authentication using NGINX](./authentication.html)",
      "url": " /security/shiroauthentication.html",
      "group": "security",
      "excerpt": "Apache Shiro is a powerful and easy-to-use Java security framework that performs authentication, authorization, cryptography, and session management. This document explains step by step how Shiro can be used for Zeppelin notebook authentication."
    }
    ,
    
  
  

    "/storage/storage.html": {
      "title": "Notebook Storage for Apache Zeppelin",
      "content"  : "<!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.-->{% include JB/setup %}# Notebook storage options for Apache Zeppelin## OverviewApache Zeppelin has a pluggable notebook storage mechanism controlled by `zeppelin.notebook.storage` configuration option with multiple implementations.There are few notebook storage systems available for a use out of the box:  * (default) use local file system and version it using local Git repository - `GitNotebookRepo`  * all notes are saved in the notebook folder in your local File System - `VFSNotebookRepo`  * storage using Amazon S3 service - `S3NotebookRepo`  * storage using Azure service - `AzureNotebookRepo`Multiple storage systems can be used at the same time by providing a comma-separated list of the class-names in the configuration.By default, only first two of them will be automatically kept in sync by Zeppelin.## Notebook Storage in local Git repository To enable versioning for all your local notebooks though a standard Git repository - uncomment the next property in `zeppelin-site.xml` in order to use GitNotebookRepo class:```  zeppelin.notebook.storage  org.apache.zeppelin.notebook.repo.GitNotebookRepo  notebook persistence layer implementation```## Notebook Storage in S3 Notebooks may be stored in S3, and optionally encrypted.  The [``DefaultAWSCredentialsProviderChain``](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html) credentials provider is used for credentials and checks the following:- The ``AWS_ACCESS_KEY_ID`` and ``AWS_SECRET_ACCESS_KEY`` environment variables- The ``aws.accessKeyId`` and ``aws.secretKey`` Java System properties- Credential profiles file at the default location (````~/.aws/credentials````) used by the AWS CLI- Instance profile credentials delivered through the Amazon EC2 metadata serviceThe following folder structure will be created in S3:```s3://bucket_name/username/notebook-id/```Configure by setting environment variables in the file **zeppelin-env.sh**:```export ZEPPELIN_NOTEBOOK_S3_BUCKET = bucket_nameexport ZEPPELIN_NOTEBOOK_S3_USER = username```Or using the file **zeppelin-site.xml** uncomment and complete the S3 settings:```  zeppelin.notebook.s3.bucket  bucket_name  bucket name for notebook storage  zeppelin.notebook.s3.user  username  user name for s3 folder structure```Uncomment the next property for use S3NotebookRepo class:```  zeppelin.notebook.storage  org.apache.zeppelin.notebook.repo.S3NotebookRepo  notebook persistence layer implementation```Comment out the next property to disable local git notebook storage (the default):```  zeppelin.notebook.storage  org.apache.zeppelin.notebook.repo.GitNotebookRepo  versioned notebook persistence layer implementation```### Data Encryption in S3#### AWS KMS encryption keysTo use an [AWS KMS](https://aws.amazon.com/kms/) encryption key to encrypt notebooks, set the following environment variable in the file **zeppelin-env.sh**:```export ZEPPELIN_NOTEBOOK_S3_KMS_KEY_ID = kms-key-id```Or using the following setting in **zeppelin-site.xml**:```  zeppelin.notebook.s3.kmsKeyID  AWS-KMS-Key-UUID  AWS KMS key ID used to encrypt notebook data in S3```In order to set custom KMS key region, set the following environment variable in the file **zeppelin-env.sh**:```export ZEPPELIN_NOTEBOOK_S3_KMS_KEY_REGION = kms-key-region```Or using the following setting in **zeppelin-site.xml**:```  zeppelin.notebook.s3.kmsKeyRegion  target-region  AWS KMS key region in your AWS account```Format of `target-region` is described in more details [here](http://docs.aws.amazon.com/general/latest/gr/rande.html#kms_region) in second `Region` column (e.g. `us-east-1`).#### Custom Encryption Materials Provider classYou may use a custom [``EncryptionMaterialsProvider``](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/EncryptionMaterialsProvider.html) class as long as it is available in the classpath and able to initialize itself from system properties or another mechanism.  To use this, set the following environment variable in the file **zeppelin-env.sh**:```export ZEPPELIN_NOTEBOOK_S3_EMP = class-name```Or using the following setting in **zeppelin-site.xml**:```  zeppelin.notebook.s3.encryptionMaterialsProvider  provider implementation class name  Custom encryption materials provider used to encrypt notebook data in S3```   ## Notebook Storage  in Azure Using `AzureNotebookRepo` you can connect your Zeppelin with your Azure account for notebook storage.First of all, input your `AccountName`, `AccountKey`, and `Share Name` in the file **zeppelin-site.xml** by commenting out and completing the next properties:```  zeppelin.notebook.azure.connectionString  DefaultEndpointsProtocol=https;AccountName=;AccountKey=  Azure account credentials  zeppelin.notebook.azure.share  zeppelin  share name for notebook storage```Secondly, you can initialize `AzureNotebookRepo` class in the file **zeppelin-site.xml** by commenting the next property:```  zeppelin.notebook.storage  org.apache.zeppelin.notebook.repo.GitNotebookRepo  versioned notebook persistence layer implementation```and commenting out:```  zeppelin.notebook.storage  org.apache.zeppelin.notebook.repo.AzureNotebookRepo  notebook persistence layer implementation```In case you want to use simultaneously your local git storage with Azure storage use the following property instead: ```  zeppelin.notebook.storage  org.apache.zeppelin.notebook.repo.GitNotebookRepo, apache.zeppelin.notebook.repo.AzureNotebookRepo  notebook persistence layer implementation```Optionally, you can specify Azure folder structure name in the file **zeppelin-site.xml** by commenting out the next property: ```   zeppelin.notebook.azure.user  user  optional user name for Azure folder structure```## Storage in ZeppelinHub  ZeppelinHub storage layer allows out of the box connection of Zeppelin instance with your ZeppelinHub account. First of all, you need to either comment out the following  property in **zeppelin-site.xml**:```<!--  zeppelin.notebook.storage  org.apache.zeppelin.notebook.repo.GitNotebookRepo, org.apache.zeppelin.notebook.repo.zeppelinhub.ZeppelinHubRepo  two notebook persistence layers (local + ZeppelinHub)-->```or set the environment variable in the file **zeppelin-env.sh**:```export ZEPPELIN_NOTEBOOK_STORAGE="org.apache.zeppelin.notebook.repo.GitNotebookRepo, org.apache.zeppelin.notebook.repo.zeppelinhub.ZeppelinHubRepo"```Secondly, you need to set the environment variables in the file **zeppelin-env.sh**:```export ZEPPELINHUB_API_TOKEN = ZeppelinHub tokenexport ZEPPELINHUB_API_ADDRESS = address of ZeppelinHub service (e.g. https://www.zeppelinhub.com)```You can get more information on generating `token` and using authentication on the corresponding [help page](http://help.zeppelinhub.com/zeppelin_integration/#add-a-new-zeppelin-instance-and-generate-a-token).",
      "url": " /storage/storage.html",
      "group": "storage",
      "excerpt": "Apache Zeppelin has a pluggable notebook storage mechanism controlled by zeppelin.notebook.storage configuration option with multiple implementations.\""
    }
    
    
  
}