Release Notes
Pivotal HD 3.0
Pivotal HD 3.0 Release Notes
Official Apache Versions in Pivotal HD 3.0
All Pivotal HD 3.0 components listed here are official Apache releases of the most recent stable versions available. Pivotal’s philosophy is to provide patches only when absolutely necessary to assure the interoperability of the components. Unless you are explicitly directed by Pivotal Support to take a patch update, each of the PHD 3.0 components needs to remain at the following package version levels to ensure a certified and supported copy of PHD 3.0.
Apache Hadoop 2.6.0
Apache HBase 0.98.4
Apache Hive 0.14.0
Hue 2.6.1
Apache Knox 0.5.0
Apache Oozie 4.1.0
Apache Pig 0.14.0
Apache Ranger 0.4.0
Apache Spark 1.2.1 (For further information about Spark, see the Spark QuickStart Guide)
Apache Tez 0.5.2
Apache ZooKeeper 3.4.6
As of PHD 3.0, the following third-party tools are deprecated, and will be removed in a future release:
Ganglia 3.5.0
Nagios 3.5.0
Ganglia Web 3.5.7
Unsupported Apache Features
The following features are shipped as part of PHD 3.0 HDFS, but are not supported:
NameNode Federation (HDFS-1052)
viewFS (HADOOP-7257)
The following Apache Components are shipped as part of PHD 3.0 YARN, but are not supported:
Fair Scheduler
MapReduce Uber AM
MapReduce Eclipse Plug-in
Behavioral Changes
In PHD 3.0, behavioral changes affect the following Hadoop components:
-
Oozie
Previous releases of the PHD Oozie client loaded the server-side configuration file, where JAVA_HOME is specific. The current version of the client does not load that server-side file. The workaround requires the following changes in its environmental setup:
The Oozie system environment requires JDK 1.6 or greater
The machine where the users will run the Oozie command line requires $JAVA_HOME to be set to version 1.6 or greater.
Tech Previews in This Release
All PHD 3.0 components listed here are official Apache releases of the most recent stable versions available. Pivotal’ philosophy is to provide patches only when absolutely necessary to assure the interoperability of the components. Unless you are explicitly directed by Pivotal Support to take a patch update, each of the PHD 3.0 components needs to remain at the following package version levels to ensure a certified and supported copy of PHD 3.0.
HDFS Encryption
YARN support for Docker
Apache Patch Information
This section describes Apache JIRAs that were addressed in PHD 3.0.
Hadoop Common/HDFS 2.6.0
PHD 3.0 provides Apache Hadoop Core 2.6.0 and the following additional Apache patches:
HDFS-3107. Introduce truncate.
HDFS-7009. Active NN and standby NN have different live nodes.
HDFS-7056. Snapshot support for truncate.
HDFS-7058. Tests for truncate CLI
HDFS-7263. Snapshot read can reveal future bytes for appended files.
HDFS-7425. NameNode block deletion logging uses incorrect appender.
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume
HDFS-7470. SecondaryNameNode need twice memory when calling reloadFromImageFile.
HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes.
HDFS-7503. Namenode restart after large deletions can cause slow processReport
HDFS-7606. Fix potential NPE in INodeFile.getBlocks().
HDFS-7634. Disallow truncation of Lazy persist files.
HDFS-7638. Small fix and few refinements for FSN#truncate.
HDFS-7643. Test case to ensure lazy persist files cannot be truncated.
HDFS-7655. Expose truncate API for Web HDFS.
HDFS-7659. Truncate should check negative value of the new length
HDFS-7676. Fix TestFileTruncate to avoid bug of HDFS-7611.
HDFS-7677. DistributedFileSystem#truncate should resolve symlinks.
HDFS-7707. Edit log corruption due to delayed block removal again.
HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.
HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure.
HDFS-7738. Revise the exception message for recover lease; add more truncate tests such as truncate with HA setup, negative tests, truncate with other operations and multiple truncates.
HDFS-7760. Document truncate for WebHDFS.
HDFS-7831. Fix the starting index and end condition of the loop in FileDiffList.findEarlierSnapshotBlocks().
HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade.
HDFS-7885. Datanode should not trust the generation stamp provided by client.
HADOOP-926. Do not fail job history iteration when encounting missing directories.
HADOOP-941: Addendum patch.
HADOOP-11321. copyToLocal cannot save a file to an SMB share unless the user has Full Control permissions.
HADOOP-11368. Fix SSLFactory truststore reloader thread leak in KMSClientProvider.
HADOOP-11381. Fix findbugs warnings in hadoop-distcp, hadoop-aws, hadoop-azure, and hadoop-openstack
HADOOP-11381. Revert. (Fix findbugs warnings in hadoop-distcp, hadoop-aws, hadoop-azure, and hadoop-openstack.)
HADOOP-11412. POMs mention "The Apache Software License" rather than "Apache License".
HADOOP-11412. Revert. (POMs mention "The Apache Software License" rather than "Apache License"
HADOOP-11490. Expose truncate API via FileSystem and shell command.
HADOOP-11509. change parsing sequence in GenericOptionsParser to parse -D parameters before -files.
HADOOP-11510. Expose truncate API via FileContext.
HADOOP-11523. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.
HADOOP-11579. Documentation for truncate.
HADOOP-11595. Add default implementation for AbstractFileSystem#truncate.
MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken service name properly.
YARN-570. Time strings are formated in different timezone.
YARN-2246. Made the proxy tracking URL always be http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections.
YARN-2571. RM to support YARN registry
YARN-2683. registry config options: document and move to core-default
YARN-2837. Support TimeLine server to recover delegation token when restarting.
YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher
YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie).
YARN-3103. AMRMClientImpl does not update AMRM token properly.
YARN-3207. Secondary filter matches entites which do not have the key being filtered for.
YARN-3227. Timeline renew delegation token fails when RM user's TGT is expired.
YARN-3239. WebAppProxy does not support a final tracking url which has query fragments and params.
YARN-3251. Fixed a deadlock in CapacityScheduler when computing absoluteMaxAvailableCapacity in LeafQueue.
YARN-3269. Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path.
HBase 0.98.4
PHD 3.0 provides HBase 0.98.4 and the following addtional Apache patches:
HBASE-212. Addendum. Fixes a unit test
HBASE-10499 In write-heavy scenario one of the regions does not get flushed causing RegionTooBusyException
HBASE-11569 Addendum for not skipping replayed edits for primary region replica
HBASE-12238 A few exceptions on startup - PARTIAL BACKPORT
HBASE-12533 staging directories are not deleted after secure bulk load
HBASE-12575 Sanity check table coprocessor classes are loadable
HBASE-12536. Reduce the effective scope of GLOBAL CREATE and ADMIN permission
HBASE-12562 Handling memory pressure for secondary region replicas - ADDENDUM for fixing findbug reported issues
HBASE-12791 HBase does not attempt to clean up an aborted split when the regionserver shutting down
HBASE-12714 RegionReplicaReplicationEndpoint should not set the RPC Codec
HBASE-12958 SSH doing hbase:meta get but hbase:meta not assigned
HBASE-13120 Allow disabling hadoop classpath and native library lookup
Hive 0.14.0
PHD 3.0 provides Hive 0.14.0 and the following additional Apache patches:
HIVE-480: HDFSCleanup thread holds reference to FileSystem
HIVE-6468: HiveServer2 (tcp mode, with SASL layer) OOMs when getting a non-thrift message
HIVE-6679: HiveServer2 should support TCP Keepalive & Server Socket Timeout on blocking calls
HIVE-7175: Provide password file option to beeline
HIVE-7270: Serde info is not shown in show create table statement, but shows in the desc table
HIVE-8295: Add batch retrieve partition objects for metastore direct sql
HIVE-8485: SUMMARY-Hive metastore NPE with Oracle DB when there is empty value for string for tblproperties/serdeproperties/etc, table not usable after creation
HIVE-8762 : HiveMetaStore.BooleanPointer should be replaced with an AtomicBoolean
HIVE-8791 : Hive permission inheritance throws exception S3
HIVE-8850: SUMMARY-[ObjectStore:: rollbackTransaction() needs to be looked into further
HIVE-8881: Receiving json "could not find job" error when web client tries to fetch all jobs from WebHCat but HDFS does not have the data.
HIVE-8888: Hive on Tez query output duplicate rows when there is explode in subqueries for joins
HIVE-8891: SUMMARY-[Another possible cause to NucleusObjectNotFoundException from drops/rollback
HIVE-8893 : Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode
HIVE-8966 : Delta files created by hive hcatalog streaming cannot be compacted
HIVE-9025 : join38.q (without map join) produces incorrect result when testing with multiple reducers
HIVE-9038: Join tests fail on Tez
HIVE-9055: Tez: union all followed by group by followed by another union all gives error
HIVE-9106 : improve the performance of null scan optimizer when several table scans share a physical path
HIVE-9112 : Query may generate different results depending on the number of reducers
HIVE-9141: HiveOnTez: mix of union all, distinct, group by generates error
HIVE-9155: HIVE_LOCKS uses int instead of bigint hive-txn-schema-0.14.0.mssql.sql
HIVE-9205/MHIVE-508: Change default Tez install directory to use /tmp instead of /user and create the directory if it does not exist
HIVE-9234: HiveServer2 leaks FileSystem objects in FileSystem.CACHE
HIVE-9235: Turn off Parquet Vectorization until all data types work: DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR
HIVE-9249: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables
HIVE-9278: Cached expression feature broken in one case
HIVE-9351: Running Hive Jobs with Tez cause templeton to never report percent complete
HIVE-9359: SUMMARY-[Hive export OOM error when table is huge. (32TB data, 4800+ partitions)]
HIVE-9382 : Query got rerun with Global Limit optimization on and Fetch optimization off
HIVE-9390: Enhance retry logic wrt DB access in TxnHandler
HIVE-9401: SimpleFetchOptimizer for limited fetches without filters
HIVE-9404 NPE in org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct()
HIVE-9436: SUMMARY-[RetryingMetaStoreClient does not retry JDOExceptions]
HIVE-9446: JDBC DatabaseMetadata.getColumns() does not work for temporary tables
HIVE-9473 : sql std auth should disallow built-in udfs that allow any java methods to be called
HIVE-9593. ORC reader should ignore new/unknown metadata streams.
HIVE-9652: STDERR redirection should not use in place updates TEZ UI
HIVE-9665: Parallel move task optimization causes race condition
HIVE-9673 : Set operationhandle in ATS entities for lookups
HIVE-9683: Client TCP keep-alive for Thrift as a workaround for THRIFT-2788
HIVE-9684: Incorrect disk range computation in ORC because of optional stream kind
HIVE-9743 Incorrect result set for vectorized left outer join
HIVE-9779 : ATSHook does not log the end user if doAs=false (it logs the hs2 server user)
HIVE-9836: Hive on Tez: fails when virtual columns are present in the join conditions (for e.g. partition columns)
HIVE-9841: IOException thrown by ORC should include the path of processing file
HIVE-9886: Tez NPE error when initialize reducer from 2 row_number function on each join side
HIVE-9892: HIVE schema failed to upgrade with schematool
HIVE-9832: Merge join followed by union and a map join in hive on tez fails.
MHIVE-479: Add input reader option to webhcat mapreduce streaming API
MHIVE-493 : change hadoop distcp command to perform directory copy and catch error logs if any
MHIVE-493 : Minor change to use destination file system while setting file status
MHIVE-493 : Use destination file system to get the destination file status
MHIVE-493 : Support MoveTask.moveFile to work with different file systems
MHIVE-527: add support to load and export between different filesystems
MHIVE-530: Removing hive.log.dir and the sortmerge noconditional task parameters from winpkg
MHIVE-542: Add hcatalog shared jars to the classpath
MHIVE-530: Remove unwanted properties from hive-site.xml of windows
PROD-559 The value of hive.zookeeper.quorum is not set in hive-site.xml
Knox 0.5.0
PHD 3.0 provides Knox 0.5.0 and the following additional Apache patches:
KNOX-492: Support service level replayBufferLimit for Ozzie, Hive and HBase.
Oozie 4.1.0
PHD 3.0 provides Apache Oozie 4.1.0 and the following additional Apache patches:
OOZIE-208. Adding missing oozie property oozie.service.HadoopAccessorService.hadoop.configurations to oozie install script.
PROD-594. Adding missing paranthesis for Function UpdateConfigWithAdditionalProperties in oozie install script.
Pig 0.14.0
PHD 3.0 provides Apache Pig 0.14.0 and the following additional Apache patches:
PIG-156: Pig command fails because the input line is too long
PIG-4334: PigProcessor does not set pig.datetime.default.tz
PIG-4342: Pig 0.14 cannot identify the uppercase of DECLARE and DEFAULT
PIG-4377: Skewed outer join produce wrong result in some cases (PIG-4377-2.patch)
PIG-4377: Skewed outer join produce wrong result in some cases
Ranger 0.4.0
PHD 3.0 provides Ranger 0.4.0 and the following additional Apache patch:
RANGER-188: Added LSB headers to Ranger Admin/Usersync init.d scripts
Tez 0.5.2
PHD 3.0 provides Apache Tez 0.5.2 and the following Apache patches:
TEZ-1642. TestAMRecovery sometimes fails.
TEZ-1775. Allow setting log level per logger.
TEZ-1800. Integer overflow in ExternalSorter.getInitialMemoryRequirement()
TEZ-1836. Provide better error messages when tez.runtime.io.sort.mb, spill percentage is incorrectly configured.
TEZ-1851. FileSystem counters do not differentiate between different FileSystems
TEZ-1852. Get examples to work in Local Mode.
TEZ-1861. Fix failing test: TestOnFileSortedOutput.
TEZ-1878. Task-specific log level override not working in certain conditions
TEZ-1924. Tez AM does not register with AM with full FQDN causing jobs to fail in some environments.
TEZ-1931. Publish tez version info to Timeline.
TEZ-1934. TestAMRecovery may fail due to the execution order is not determined.
TEZ-1942. Number of tasks show in Tez UI with auto-reduce parallelism is misleading.
TEZ-1949: Whitelist TEZ_RUNTIME_OPTIMIZE_SHARED_FETCH for broadcast edges
TEZ-1962. Fix a thread leak in LocalMode.
TEZ-1990. Tez UI: DAG details page shows Nan for end time when a DAG is running.
TEZ-2024. Compiliation error due to conflict.
TEZ-2024. TaskFinishedEvent may not be logged in recovery.
TEZ-2031. Tez UI: horizontal scrollbars do not appear in tables, causing them to look truncated.
TEZ-2037. Should log TaskAttemptFinishedEvent if taskattempt is recovered to KILLED
TEZ-2038. TEZ-UI DAG is always running in tez-ui when the app is failed but no DAGFinishedEvent is logged.
TEZ-2043. Tez UI: add progress info from am webservice to dag and vertex views.
TEZ-2052. Tez UI: log view fixes, show version from build, better handling of ats url config.
TEZ-2056. Tez UI: fix VertexID filter,show only tez configs by default,fix appattemptid.
TEZ-2063. Tez UI: Flaky log url in tasks table.
TEZ-2065. Setting up tez.tez-ui.history-url.base with a trailing slash can result in failures to redirect correctly.
TEZ-2068. Tez UI: Dag view should use full window height, disable webuiservice in localmode.
TEZ-2069. Tez UI: appId should link to application in dag details view.
TEZ-2077. Tez UI: No diagnostics on Task Attempt Details page if task attempt failed
TEZ-2078. Tez UI: Task logs url use in-progress url causing various errors.
TEZ-2079. Tez UI: trailing slash in timelineBaseUrl in ui should be handled.
TEZ-2092. Tez UI history url handler injects spurious trailing slash.
TEZ-2098. Tez UI: Dag details should be the default page for dag, fix invalid time entries for failed Vertices.
TEZ-2101. Tez UI: Issues on displaying a table.
TEZ-2102. Tez UI: DAG view has hidden edges, dragging DAG by holding vertex causes unintended click.
TEZ-2106. TEZ UI: Display data load time, and add a refresh button for items that can be refreshed.
TEZ-2112. Tez UI: fix offset calculation, add home button to breadcrumbs.
TEZ-2114. Tez UI: task/task attempt status is not available when its running.
TEZ-2116. Tez UI: dags page filter does not work if more than one filter is specified.
TEZ-2134. TEZ UI: On request failure, display request URL and server name in error bar.
TEZ-2135. ACL checks handled incorrectly in AMWebController.
TEZ-2136. Some enhancements to the new Tez UI.
TEZ-2142. TEZ UI: Breadcrumb border color looks out of place in wrapped mode.
TEZ-2158. TEZ UI: Display dag/vertex names, and task/attempt index in breadcrumb
TEZ-2160. Tez UI: App tracking URL should support navigation back.
TEZ-2165. Tez UI: DAG shows running status if killed by RM in some cases.
ZooKeeper 3.4.6
PHD 3.0 provides ZooKeeper 3.4.6 and the following additional Apache patches:
ZOOK-46: Port ZooKeeper-1506 that handles connection issues by retrying DNS
Fixed in PHD 3.0
The following features and fixes were contributed back to Apache with the release of PHD 3.0:
Pivotal Bug No. |
Apache JIRA |
Description |
Component |
|---|---|---|---|
BUG-28115 |
HADOOP-11321 |
CopyToLocal cannot save a file to an SMB share unless the user has full Control permissions |
Hadoop Common |
BUG-32505 |
HADOOP-11368 |
Fix SSLFactory truststore reloader thread leak in KMSClientProvider |
HDFS |
BUG-28620 |
HADOOP-9629 |
Synchronize WASB code from upstream Apache |
Hadoop Common |
BUG-33034 |
HBASE-12238 |
Exceptions on startup |
HBase |
BUG-26389 |
HBASE-12575 |
Botched alter table involving coprocessor crashes RS |
HBase |
BUG-29996 |
HBASE-12791 |
HBase does not attempt to clean up an aborted SPLIT when the Regions Server is shutting down |
HBase |
BUG-31373 |
HBASE-12958 |
SSH doing hbase:meta displays "hbase:meta not assigned" message |
HBase |
BUG-32330 |
HBASE-13120 |
Allow native library path to be overriden |
HBase |
BUG-32106 |
HDFS-7009 |
Active NN and standby NN have different live nodes |
HDFS |
BUG-28206 |
HDFS-7425 |
NameNode block deletion logging uses incorrect appender. |
HDFS |
BUG-29862 |
HDFS-7489 |
Incorrect locking in FsVolumeList#checkDirs can hang datanodes. |
HDFS |
BUG-29478 |
HDFS-7560 |
ACLs removed by removeDefaultAcl() will be back after NameNode restart/failover |
HDFS |
BUG-31221 |
HDFS-7707 |
Edit log corruption due to delayed block removal again |
HDFS |
BUG-31411 |
HDFS-7714 |
Simultaneous restart of HA NameNodes and DataNode can causes DataNode to register successfully with only one NameNode. |
HDFS |
BUG-31164 |
HDFS-7733 |
NFS: readdir/readdirplus returns null directory attribute on failure |
HDFS |
BUG-32197 |
HDFS-7843 |
Truncate test failed during rollback |
HDFS |
BUG-33108 |
HDFS-7925 |
HDFS truncate command fails |
HDFS |
BUG-30864 |
HIVE-5700, HIVE-9445 |
Revert HIVE-5700 - enforce single date format for partition column storage |
Hive |
BUG-29572 |
HIVE-7270 |
Serde info is not shown in SHOW CREATE TABLE statement, but shows in the desc table |
Hive |
BUG-27598 |
HIVE-8295 |
DirectSQL with Oracle metastore fails with error ORA-01795, Maximum number of expression can not be more than 1000 |
Hive |
BUG-29164 |
HIVE-8870 |
Errors when selecting a struct field within an array from ORC based tables |
Hive |
BUG-27856 |
HIVE-8881 |
Receiving json error when web client tries to fetch all jobs from webhcat where HDFS does not have the data. |
WebHCat |
BUG-28232 |
HIVE-8888 |
Hive on Tez query output duplicate rows when JOIN subqueries are expanded |
Hive |
BUG-30663 |
HIVE-8893, HIVE-9473 |
UDF that allows untrusted code execution should be disabled |
Hive |
BUG-26804 |
HIVE-8948 |
Fix or disable flaky test testTransactionBatchEmptyCommit |
Hive |
BUG-29631 |
HIVE-8966 |
Delta files created by hive hcatalog streaming cannot be compacted |
Hive |
BUG-30288 |
HIVE-9025, HIVE-9112 |
Incorrect result due to bug in constant propagation optimizer |
Hive |
BUG-29161 |
HIVE-9051 |
TezJobMonitor in-place updates logs too often to logfile |
Hive |
BUG-29158 |
HIVE-9067 |
OrcFileMergeOperator may create merge file that does not match properties of input files |
Hive |
BUG-29162 |
HIVE-9090 |
Rename "Tez File Merge Work" to smaller name |
Hive |
BUG-29159 |
HIVE-9111 |
Potential NPE in OrcStruct for list and map types |
Hive |
BUG-29840 |
HIVE-9249 |
java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables |
Hive |
BUG-30063 |
HIVE-9278 |
Cached expression feature broken in one case |
Hive |
BUG-30305 |
HIVE-9359 |
Hive export OOM error when table is huge. (32TB data, 4800+ partitions) |
Hive |
BUG-31213 |
HIVE-9593 |
ORC reader should ignore new/unknown metadata streams |
Hive |
BUG-32081 |
HIVE-9652 |
STDERR redirection should not use in-place updates TEZ UI |
Hive |
BUG-31288 |
HIVE-9665 |
ParallelMoveTask test failed on Ambari-deployed cluster due to OOM error |
Hive |
BUG-32089 |
HIVE-9743 |
Incorrect result set for left outer join when executed with tez versus mapreduce |
Hive |
BUG-32751 |
HIVE-9841 |
IOException thrown by ORC should include the path of processing file |
Hive |
BUG-29423 |
HADOOP-799 |
NullPointerException in FSDataset |
MapReduce |
BUG-28571 |
HBASE-12533 |
Staging directories are not deleted after secure bulk load |
HBase |
BUG-29367 |
HDFS-7443 |
Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume |
HDFS |
BUG-30397 |
HDFS-7470 |
SecondaryNameNode need twice memory when calling reloadFromImageFile |
HDFS |
BUG-27347 |
HDFS-7503 |
Namenode restart after a large deletion could cause slow processReport. |
HDFS |
BUG-28815 |
HIVE-6679 |
HiveServer2 should support TCP Keepalive & Server Socket Timeout on blocking calls |
Hive |
BUG-32184 |
HIVE-9779 |
ATSHook does not log the end user if doAs=false (it logs the hs2 server user) |
Hive |
BUG-32273 |
n/a |
Unable to change Hue users' passwords via the Hue UI |
Hue |
BUG-32998 |
n/a |
Fix error on deleting users when upgrading Hue |
Hue |
BUG-32389 |
n/a |
Users cannot use the Job Browser to investigate failed jobs |
Hue |
BUG-31139 |
n/a |
Fix django-db.utils.DatabaseError: current transaction is aborted when setting up Hue on PostgreSQL |
Hue |
BUG-28353 |
KNOX-464 |
Knox not returning proper redirection url when executing file creation/upload operation via load balancer |
Knox |
BUG-31968 |
OOZIE-2119 |
Oozie Distcp action fails with error : Returned value from distcp is non-zero (-1) |
Oozie |
BUG-30095 |
RANGER-192 |
User Detail Page hangs if user has many groups |
Ranger |
BUG-32094 |
TEZ-2135 |
ACL checks working incorrectly in AMWebController |
Tez |
BUG-29254 |
YARN-2964 |
RM prematurely cancels tokens for jobs that submit jobs (oozie) |
YARN |
BUG-31944 |
YARN-3239 |
WebAppProxy in YARN broken for redirecting to the Tez View in Ambari |
YARN |
BUG-29055 |
YARN-570 |
Time strings are formated in different timezone |
YARN |
BUG-20353 |
YARN-570 |
Resource Manager Timezone only appears in GMT, should appear in Local Time to Browser |
YARN |
BUG-31195 |
KNOX-473 |
Knox fails to rewrite Load balancer URI when doing out request |
Knox |
BUG-30967 |
KNOX-491 |
Increase default replay buffer size to 8K |
Knox |
BUG-30968 |
KNOX-492 |
Support service level replayBufferLimit for Oozie, Hive and HBase |
Knox |
BUG-28636 |
PIG-4334 |
PigProcessor does not set pig.datetime.default.tz |
Pig |
BUG-31704 |
RANGER-242 |
Ranger on Ambari: Ranger config migration script fails to migrate ranger-admin/install.properties configs to Ranger config properties under Ambari. |
Ranger |
BUG-29175 |
TEZ-1775 |
Allow setting log level per logger (See also TEZ-1878) |
Tez |
BUG-29373 |
TEZ-1851 |
FileSystem counters do not differentiate between different FileSystems |
Tez |
BUG-31540 |
TEZ-2024, TEZ-2037, TEZ-2087 |
Recovered Task/TaskAttempt status is not available on tez-ui |
Tez |
BUG-31640 |
TEZ-2029 |
Issues on DAG's view |
Tez |
BUG-31481 |
TEZ-2038 |
DAG is always in running status in tez-ui when the app is failed before DAG is completed |
Tez |
BUG-31626 |
TEZ-2050, TEZ-2114 |
No Running Status shows for Task/TaskAttempt |
Tez |
BUG-31637 |
TEZ-2094, YARN-3207 |
Running Task/TaskAttempt are not excluded when choosing other status in the Dropdownlist |
Tez, YARN |
BUG-27488 |
YARN-2837 |
Timeline server needs to recover the timeline DT when restarting |
YARN |
BUG-32619 |
YARN-3227 |
HiveServer2 long running test is failing due to TezSession not being able to renew TIMELINE_DELEGATION_TOKEN |
YARN |
Known Issues for Pivotal HD 3.0
Key |
Apache JIRA |
Summary |
Component/s |
|---|---|---|---|
RMP-2381 |
|
Sub-5-Second: Lookup Query Metastore in less than 200 milliseconds (tech preview) |
Hive |
RMP-2377 |
HIVE-7926 |
Sub-5-Second: LLAP Daemons and LLAP API (tech preview) |
Hive |
RMP-3332 |
HIVE-9792 |
RMP-2430 SQL: Partition pruning support for Intervals |
Hive |
RMP-3333 |
HIVE-9792 |
RMP-2430 SQL: ORC PPD support for Intervals |
Hive |
RMP-2448 |
|
Sub-5-Second: Umbrella: Execute TPC-DS query 55 in under 5 second at 30TB and Query 27 at 1 TB (tech preview) |
Hive |
RMP-2662 |
|
Sub-Second: Vectorization enhancements - MapJoin |
Hive |
RMP-2620 |
|
CBO: Introduce semi join reduction in Calcite |
Hive |
RMP-2449 |
HIVE-9675 |
Transactions: Support BEGIN, COMMIT, ROLLBACK |
Hive |
RMP-2634 |
|
Replication for Hive (partial) |
Hive |
RMP-2430 |
HIVE-9792 |
SQL: Implement Interval types and datetime arithmetic (partial) |
Hive |
BUG-26944 |
|
E0803: IO error, null during scheduling feed on a cluster with wire enc on |
Oozie |
BUG-33113 |
|
HBase-HA master fail over immediately |
HBase |
BUG-31943 |
|
test_IntegrationTestRegionReplicas region not serving |
HBase |
BUG-32509 |
|
test_runIntegrationTestMTTRwithDistributedLogs due to failure in testKillRsHoldingMeta |
HBase |
BUG-31353 |
|
IntegrationTestRegionReplicaPerf region not serving |
HBase |
BUG-29903 |
|
IntegrationTestMTTR slowness |
HBase |
BUG-31936 |
|
test_hbaseBigLinkedList[Loop] TimeoutException |
HBase |
BUG-27858 |
|
HBase HA tests fail with Table Namespace Manager not ready yet |
HBase |
BUG-27509 |
HBASE-12464 |
test_ReplicaCopyTable_verifyCopyTable[tableone] fails after trying several times for 'Failed to find location, tableName=hbase:meta, row=' |
HBase |
BUG-27256 |
|
Create mr based Hbase Long running application for Rollingupgrade |
HBase |
BUG-25961 |
|
HBase tests failing with org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out; 10000ms |
HBase |
BUG-23619 |
|
On Windows, tests fail complaining about Cannot run program /usr/bin/env |
HBase |
BUG-23981 |
|
IOException: Invalid HFile block magic in Windows unit tests |
HBase |
BUG-22841 |
|
HBase test failed with Insufficient permissions (user=gpadmin@example.com error |
HBase |
BUG-21254 |
|
hbck tool fixMeta flag not able to recreate deleted entries from hbase:meta table when just one column is deleted |
HBase |
BUG-23897 |
|
HBase REST server continuously restarting in secure windows env |
HBase |
BUG-33278 |
HBASE-13239 |
HBASE grants at specific column level does not work for Groups |
HBase |
BUG-27656 |
HBASE-12472, HBASE-13192 |
IntegrationTestBulkLoad fails on mapreduce |
HBase |
BUG-27756 |
|
test_IntegrationTestMultiGetRegionReplicas slowness |
HBase |
BUG-31822 |
|
hcat startup failed on windows |
HCatalog |
BUG-23590 |
HIVE-8247 |
Pig cursor written to Hive via HCat doesn't NULL-fill missing columns |
HCatalog |
BUG-27971 |
HDFS-7558 |
HDFS daemons JVM pauses when the NFS client and server on the same host as other HDFS daemons |
HDFS |
BUG-25352 |
|
Exit code 0 when command (put/copying a file bigger than quota) fails. |
HDFS |
BUG-31624 |
|
Dropping a 2000+ partition table takes over 10 minutes |
Hive |
BUG-32416 |
|
Hive result mismatch in decimal digits when converting from double |
Hive |
BUG-31671 |
|
HS2 concurr tests intermittently fail with http mode on secure Openstack throwing org.apache.hive.service.auth.HttpAuthenticationException: Kerberos authentication failed |
Hive |
BUG-28097 |
HIVE-8915 |
Log file explosion due to non-existence of COMPACTION_QUEUE table |
Hive |
BUG-30984 |
|
Hive compaction tests failing |
Hive |
BUG-33167 |
|
Hive does not prevent non whitelisted config params from being set under certain condtions |
Hive |
BUG-32795 |
|
HIVE schema failed to upgrade with schematool |
Hive |
BUG-33168 |
|
hive authorization checks on configs should not be done if authorization is diabled |
Hive |
BUG-28218 |
|
Hive CLI hangs |
Hive |
BUG-27636 |
|
Oracle: intemittent acid_concurrency test failures due to NoSuchLockException |
Hive |
BUG-26243 |
|
CBO : Q51 fails in explain with Failed to breakup Windowing invocations into Groups. |
Hive |
BUG-27507 |
|
Hive CLI returns 0 even in case of failures |
Hive |
BUG-27592 |
|
Beeline Client Connections Errors - Obscure errors and exception swallow - 4 |
Hive |
BUG-29427 |
|
Turn off Parquet Vectorization until all data types work: DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR |
Hive |
BUG-27582 |
|
Beeline Client Connections Errors - Obscure errors and exception swallow - 1 |
Hive |
BUG-30022 |
HIVE-7693 |
Invalid column ref error in order by when using column alias in select clause and using having |
Hive |
BUG-25058 |
HIVE-8459 |
DbLockManager reader locks are locking tables in addition to partitions |
Hive |
BUG-25064 |
|
Hive : Q51 fails in CBO with "Failed to breakup Windowing invocations into Groups. At least 1 group must only depend on input columns" |
Hive |
BUG-23260 |
|
BUG-22878 insert after alter table issue |
Hive |
BUG-23830 |
HIVE-8263 |
CBO : Query query has date_dim d1,date_dim d2 and date_dim d3 but the explain has d1, d1 and d1 |
Hive |
BUG-21468 |
|
With CBO enabled queries fail with Failed to breakup Windowing invocations into Groups |
Hive |
BUG-20856 |
|
Column names are missing from join expression in Map join with CBO enabled |
Hive |
BUG-18450 |
HIVE-7160 |
Vectorization Udf: GenericUDFConcat, is not supported for String+Long |
Hive |
BUG-23874 |
|
Map join sizing done by auto.convert.join.noconditionaltask.size doesn't take into account Hash table overhead and results in OOM |
Hive |
BUG-25424 |
|
When table is renamed stats are lost as changes are not propagated to metastore tables TAB_COL_STATS and PART_COL_STATS |
Hive |
BUG-27637 |
|
Postgres: intermittent test failure due to "ERROR: could not serialize access due to read/write dependencies among transactions" |
Hive |
BUG-33338 |
|
Hiveserver2 in HTTP mode is not applying auth_to_local rules |
Hive |
BUG-32859 |
HIVE-9977 |
Hive metastore auto major compaction not happening if data is loaded through dynamic partitioning insert |
Hive |
BUG-33321 |
|
Created tables not available under Hcatalog |
Hue |
BUG-28427 |
|
NPE in ambari knox install |
Knox |
BUG-27094 |
|
Knox HBase tests skipped for ambari test config |
Knox |
BUG-27091 |
|
Knox YarnSysTest skipped in secure mode for ambari test config |
Knox |
BUG-27089 |
|
Knox YarnSysTest skipped in secure mode for gsInstaller test config |
Knox |
BUG-28039 |
|
Knox Failed Authentication Auditing |
Knox |
BUG-28031 |
|
Knox rolling upgrade system test support for wire encryption |
Knox |
BUG-27802 |
|
Improve reliability of Knox smoke test in system test deployment |
Knox |
BUG-27906 |
|
Increase timeout of system test WebHCatDslSysTest-testPig |
Knox |
BUG-27549 |
|
Rolling upgrade: knox - webHbase smoke test failed after upgrading core components |
Knox |
BUG-25049 |
|
Two WebHdfsHaFuncTest are unstable and should be fixed and enabled |
Knox |
BUG-23889 |
|
Windows security scripting should setup Knox to use ActiveDirectory for LDAP |
Knox |
BUG-32419 |
|
TestBundleVersion: oozie bundle takes 1 hr 10mins to recognized a three completed/killed coordinator/wf actions |
Oozie |
BUG-32763 |
|
yum cannot find Xvfb for centos06 |
Oozie, RelEng |
BUG-32690 |
|
test_pig_via_oozie_ha[resourcemanager] exception:NotFoundException,message:java.lang.Exception: job, job_1425457888074_0001, is not found,javaClassName |
Oozie |
BUG-31426 |
|
Oozie failed to start due to "SA" schema not created |
Oozie, RelEng |
BUG-32026 |
|
test_mapred_via_oozie_ha[oozie-nn-rm-5-jobs] FAILED |
Oozie |
BUG-31349 |
|
oozie job in running state is causing intermittent timeout failures |
Oozie |
BUG-33180 |
|
oozie logging tests fail on MR framework with postgres 9 |
Oozie |
BUG-26766 |
|
[Rolling Upgrade] Allow Prepare Oozie.war to be done without stopping Oozie Server |
Oozie |
BUG-28614 |
|
Oozie property oozie.authentication.kerberos.principal is being used for authentication of webhdfs endpoint |
Oozie |
BUG-24517 |
|
oozie needs to support Application Tag |
Oozie |
BUG-23655 |
|
Oozie stuck on mapreduce job because workflow state isn't changing from RUNNING to KILLED |
Oozie |
BUG-24118 |
|
PHD Windows: Update/clean up oozie's SQL Server JDBC driver copies |
Oozie |
BUG-23527 |
|
Oozie config referred to versioned hadoop config |
Oozie |
BUG-26428 |
|
XaAdmin and XaAgent jobs for Debian failing on Unix clusters. |
Ranger |
BUG-31949 |
|
In hive plugin TABLE policy restriction for all tables {{*}} fails when UDF policy for all functions {{*}} is maintained for a database. |
Ranger |
BUG-31749 |
|
Usersync NPE when object does not have userNameAttribute |
Ranger |
BUG-31782 |
|
Ranger gives error when using hyphen or space in lastname field |
Ranger |
BUG-33056 |
|
{start,stop}-thriftserver equivalent script on Windows is missing |
Spark |
BUG-33054 |
|
Spark examples are packaged differently on Windows |
RelEng, Spark |
BUG-33335 |
|
spark make-distribution.sh doesn't evaluate SPARK_HIVE on OSX |
Spark |
BUG-32342 |
|
Minimum/Maximum Duration show a empty bracket next to 0 secs when you purposefully failed a job. |
Tez |
BUG-31522 |
|
You cannot search by the name of 'Queue' |
Tez |
BUG-31526 |
|
The order of columns will go back to the initial setting after you refresh a page or come back from another page. |
Tez |
BUG-31598 |
|
Tez UI vertical scroll broken from time to time |
Tez |
BUG-31594 |
|
TEZ view UI Column selector should have a select all or something to make selecting columns easier |
Tez |
BUG-31565 |
|
When you kill a job in the middle of running it, it Diagnostics shows 'killed/failed due to:null". |
Tez |
BUG-26419 |
|
Tez application fails with diff client and AM version |
Tez |
BUG-27562 |
|
WordCount on Tez with non-nightly failed with NPE |
Tez |
BUG-31583 |
|
Tez UI : DAG fails but Task attempts don't show any failues |
Tez |
BUG-31602 |
|
Tez UI :Task Tab should have Vertex name as Hyper link not Vertex ID |
Tez |
BUG-31595 |
|
Tez UI : Per DAG counters not getting aggregated |
Tez |
BUG-31623 |
|
Tez UI : Increase the size of table name tex box in the DAG view |
Tez |
BUG-24831 |
|
TEZ windows test framework needs to respect ${fs.defaultFS} in tez configs. |
Tez |
BUG-25194 |
|
AM running out of memory for partitioned bucketed tables |
Tez |
BUG-25380 |
|
For hive to work with Virtual columns and parittion names, we need information available within a record-reader |
Tez |
BUG-24518 |
|
Tez needs to support Application Tag |
Tez |
BUG-29279 |
|
Distributedshell App fails after unregistering the application |
YARN |
BUG-27194 |
|
[YARN-2821] Failures encountered on Distributed Shell on windows |
YARN |
BUG-26907 |
HADOOP-11274 |
[HADOOP-11274] ConcurrentModificationException in Configuration Copy Constructor |
YARN |
BUG-27590 |
|
Distributed shell job does not execute python script in Windows |
YARN |
BUG-26875 |
|
Sometimes NodeManager keeps in tracking non-exitant completed container process |
YARN |
BUG-22522 |
|
MR Job failed when run via oozie and NN and RM were being killed |
YARN |
BUG-33546 |
YARN-3393 |
Getting application(s) goes wrong when app finishes before starting the attempt |
YARN |
BUG-27632 |
|
Add feature of retry in zookeeper client |
Zookeeper |
BUG-30784 |
ZOOKEEPER-1952 |
zookeeper.log.file property is not respected; log output goes only to the zookeeper.out |
Zookeeper |
BUG-19255 |
|
Uninstalling the ZooKeeper RPM does not remove all ZooKeeper directories |
Zookeeper |