azkaban

azkaban 是一个用Java开发的开源调度工具workflow,同类型的工具有用Python开发的 airflow 和 luigi.

build

在我的windows上无法build成功，我在CentOS上build成功了

git clone https://github.com/azkaban/azkaban.gitcd azkaban./gradlew build installDist

run

我的需求比较简单 solo server 就可以满足要求

cd azkaban-solo-server/build/install/azkaban-solo-serverbin/azkaban-solo-start.shbin/azkaban-solo-shutdown.sh

config

使用MySQL数据库，MySQL新建用户

CREATE DATABASE azkaban;CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';GRANT ALL PRIVILEGES ON azkaban.* to 'azkaban'@'%';

修改配置文件

#设置为上海时间(东八区)，否则会按美国时间执行default.timezone.id=Asia/Shanghai#注释掉默认的H2数据库配置后，配置MySQL数据库database.type=mysqlmysql.port=3306mysql.host=192.168.0.1mysql.database=azkabanmysql.user=usernamemysql.password=passwordmysql.numconnections=10#配置告警邮件mail.sender=xxx@163.commail.host=smtp.163.commail.user=mailUsernamemail.password=mailPassword

启动的时候会自动创建数据库

plugin

使用ant编译，需要安装ant

sudo yum install antsudo yum install ant-junitsudo apt-get install antsudo apt-get install ant-optional

git clone https://github.com/azkaban/azkaban-plugins.git# step 1cd azkaban-pluginscd pluginscd hadoopsecuritymanagerant package# step 2cd ..cd cryptoant package# step 3cd ..cd jobtypeant package

azkaban-plugins/dist/jobtype/packages/azkaban-jobtype-3.0.0.tar.gz

解压到azkaban/azkaban-solo-server/build/install/azkaban-solo-server/plugins/jobtypes

commonprivate.properties

common.properties

execute.as.user 设置为false，否者会报错

# configs for jobtype security settingsexecute.as.user=falseazkaban.native.lib=

我目前只想运行java job，不想使用hadoop，修改源代码，可以不使用hadoop。否则没有配置hadoop 环境会报错。

--- a/plugins/jobtype/src/azkaban/jobtype/JavaJob.java+++ b/plugins/jobtype/src/azkaban/jobtype/JavaJob.java@@ -75,8 +75,8 @@ public class JavaJob extends JavaProcessJob {     classPath.add(getSourcePathFromClass(JavaProcessJob.class));     classPath.add(getSourcePathFromClass(SecurityUtils.class)); -    classPath.add(HadoopConfigurationInjector.getPath(getJobProps(),-        getWorkingDirectory()));+    //classPath.add(HadoopConfigurationInjector.getPath(getJobProps(),+    //    getWorkingDirectory()));      String loggerPath = getSourcePathFromClass(org.apache.log4j.Logger.class);     if (!classPath.contains(loggerPath)) {@@ -157,8 +157,8 @@ public class JavaJob extends JavaProcessJob {    @Override   public void run() throws Exception {-    HadoopConfigurationInjector.prepareResourcesToInject(getJobProps(),-        getWorkingDirectory());+    //HadoopConfigurationInjector.prepareResourcesToInject(getJobProps(),+    //    getWorkingDirectory());     super.run();   } }--- a/plugins/jobtype/src/azkaban/jobtype/JavaJobRunnerMain.java+++ b/plugins/jobtype/src/azkaban/jobtype/JavaJobRunnerMain.java@@ -104,7 +104,7 @@ public class JavaJobRunnerMain {       }       _logger.info("Class name " + className); -      HadoopConfigurationInjector.injectResources(new Props(null, props));+      //HadoopConfigurationInjector.injectResources(new Props(null, props));        // Create the object using proxy       if (SecurityUtils.shouldProxy(props)) {

重新编译打包就可以