Hive MetaException Invalid partition key & values
在 drop 某个 Table 的时候报了以下异常信息:
hive> drop table temp.temp_log_day;
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Invalid partition key & values; keys [dt, device_type, province_alias, hour, ], values [20180312, ])
原因是在 Hive 的元数据库中记录该 Table 还有未 drop 掉的 Partition,但是在 HDFS 上已经不存在该 Partition 的路径了。按照提示信息先 drop 对应的 Partition。drop Partition 的时候会提示具体的路径信息:
hive> ALTER TABLE temp.temp_log_day DROP PARTITION (dt='20180312');
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Failed to delete parent: File does not exist: /user/hive/warehouse/temp.db/temp_log_day/dt=20180312/device_type=box/province_alias=00
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getContentSummaryInt(FSDirStatAndListingOp.java:492)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getContentSummary(FSDirStatAndListingOp.java:139)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getContentSummary(FSNamesystem.java:3928)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getContentSummary(NameNodeRpcServer.java:1200)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getContentSummary(ClientNamenodeProtocolServerSideTranslatorPB.java:877)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
先在 HDFS 上创建对应的目录:
$ hadoop fs -mkdir -p /user/hive/warehouse/temp.db/temp_log_day/dt=20180312/device_type=box/province_alias=00
再次执行 drop 操作:
hive> drop table temp.temp_log_day;
Moved: 'hdfs://ycluster/user/hive/warehouse/temp.db/temp_log_day' to trash at: hdfs://ycluster/user/test/.Trash/Current
OK