Creating MySQL user ranger2 failed

1
2
2021-07-01 18:44:53,480  [E] Creating MySQL user ranger2 failed..
2021-07-01 18:44:53,496 [E] DB schema setup failed! Please contact Administrator.

原因是Ranger安装程序默认会使用MySQL的root账号创建所需要的账号,也就是DBA过程。如果不需要安装程序自动创建,则将install.properties中的以下配置前的#号去掉:

阅读全文 »

如果变量值中包含斜杠(/)特殊字符,在使用sed命令的做行内字符串替换时可以使用井号(#)做为sed语法分隔符,如下:

1
2
GITLAB_PROJECT_CLONE_URL=ssh://git@192.168.1.10:50022/test/Documentation.git
sed -i "11s#GITLAB_PROJECT_CLONE_URL#$GITLAB_PROJECT_CLONE_URL#" config.xml

shell -c {string}:表示命令从-c后的字符串读取。在需要使用管道或者重定向需要sudo时很有用,如下:

1
2
$ sudo find ../*/exportFiles -mtime +15 -name "*" | xargs -I {} rm -rf {}
rm: cannot remove `i_20201108_CONTENTINFO_xy_001.dat': Permission denied

按照以下方式处理即可:

阅读全文 »

异常信息如下:

1
2
3
4
5
org.apache.ranger.plugin.client.HadoopException: Unable to connect to Hive Thrift Server instance..
Unable to connect to Hive Thrift Server instance..
Could not open client transport with JDBC Uri: jdbc:hive2://192.168.72.212:10000: Could not establish connection to jdbc:hive2://192.168.72.212:10000: Required field 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null, configuration:{set:hiveconf:hive.server2.thrift.resultset.default.fetch.size=1000, use:database=default}).
Could not establish connection to jdbc:hive2://192.168.72.212:10000: Required field 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null, configuration:{set:hiveconf:hive.server2.thrift.resultset.default.fetch.size=1000, use:database=default}).
Required field 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null, configuration:{set:hiveconf:hive.server2.thrift.resultset.default.fetch.size=1000, use:database=default}).

百度后是因为Hive版本不匹配导致的。我的测试环境Hive版本是2.1.1,通过下面的命令发现确实版本不一致:

阅读全文 »

告警信息及如何解决见我的另外一篇博客:MySQL JDBC连接异常:javax.net.ssl.SSLException: closing inbound before receiving peer’s close_notify

但是Ranger的安装配置install.properties中未启用ssl,配置信息如下:

1
2
3
4
5
6
7
8
9
10
#SSL config
db_ssl_enabled=false
db_ssl_required=false
db_ssl_verifyServerCertificate=false
#db_ssl_auth_type=1-way|2-way, where 1-way represents standard one way ssl authentication and 2-way represents mutual ssl authentication
db_ssl_auth_type=2-way
javax_net_ssl_keyStore=
javax_net_ssl_keyStorePassword=
javax_net_ssl_trustStore=
javax_net_ssl_trustStorePassword=
阅读全文 »

关于Iptables的介绍个人强烈推荐:http://www.zsythink.net/archives/tag/iptables/page/2/。这位兄弟介绍的很详细。

我个人的需求是在同一个网络内从网络上把测试主机与正式环境主机间的网络进行隔离。我的思路是采用OUTPUT filter表,采用黑名单方式禁用正式环境网络的访问。如果确实需要访问正式环境的某个IP和端口,则添加特殊规则放行。根据Iptables的规则匹配顺序原则,放行特定IP和端口的规则要在网段禁用规则之前。以下是我的名列列表:

1
2
3
4
5
iptables -t filter -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -t filter -I OUTPUT -d 192.168.72.0/24 -j DROP
iptables -t filter -I OUTPUT -d 192.168.72.1/32 -j ACCEPT
service iptables save
阅读全文 »

异常信息如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Thu Nov 26 23:54:22 CST 2020 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Thu Nov 26 23:54:23 CST 2020 WARN: Caught while disconnecting...

EXCEPTION STACK TRACE:



** BEGIN NESTED EXCEPTION **

javax.net.ssl.SSLException
MESSAGE: closing inbound before receiving peer's close_notify

STACKTRACE:

javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify
at sun.security.ssl.Alert.createSSLException(Alert.java:133)
at sun.security.ssl.Alert.createSSLException(Alert.java:117)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:270)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:261)
at sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:656)
at sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:635)
at com.mysql.jdbc.MysqlIO.quit(MysqlIO.java:2246)
at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4201)
at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1472)
at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:353)
at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:120)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:483)
at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:296)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
at org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
at org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:420)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:821)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:338)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:217)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:515)
at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:544)
at org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:399)
at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:336)
at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:297)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:599)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:564)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:626)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:416)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6490)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:238)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1652)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3367)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3406)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3386)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3640)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:236)
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:221)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:366)
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:310)
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:290)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:266)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:558)
at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)


** END NESTED EXCEPTION **

解决方法:在JDBC连接串中添加useSSL=false配置,如:jdbc:mysql://192.168.72.212/hive?createDatabaseIfNotExist=true&amp;characterEncoding=utf-8&amp;useSSL=false

阅读全文 »

  • # 号截取:删除左边字符,保留右边字符。
  • ## 号截取:删除左边字符,保留右边字符。
  • %号截取:删除右边字符,保留左边字符
  • %% 号截取:删除右边字符,保留左边字符

示例一:

1
2
3
4
5
6
7
8
9
$ var='abcbd'
$ echo ${var#*b}
cbd
$ echo ${var##*b}
d
$ echo ${var%b*}
abc
$ echo ${var%%b*}
a
阅读全文 »
0%