Release Notes

Release Notes

Pivotal HD 3.0

Pivotal HD 3.0 Release Notes

Official Apache Versions in Pivotal HD 3.0

All Pivotal HD 3.0 components listed here are official Apache releases of the most recent stable versions available. Pivotal’s philosophy is to provide patches only when absolutely necessary to assure the interoperability of the components. Unless you are explicitly directed by Pivotal Support to take a patch update, each of the PHD 3.0 components needs to remain at the following package version levels to ensure a certified and supported copy of PHD 3.0.

  • Apache Hadoop 2.6.0

  • Apache HBase 0.98.4

  • Apache Hive 0.14.0

  • Hue 2.6.1

  • Apache Knox 0.5.0

  • Apache Oozie 4.1.0

  • Apache Pig 0.14.0

  • Apache Ranger 0.4.0

  • Apache Spark 1.2.1 (For further information about Spark, see the Spark QuickStart Guide)

  • Apache Tez 0.5.2

  • Apache ZooKeeper 3.4.6

As of PHD 3.0, the following third-party tools are deprecated, and will be removed in a future release:

  • Ganglia 3.5.0

  • Nagios 3.5.0

  • Ganglia Web 3.5.7

Unsupported Apache Features

The following features are shipped as part of PHD 3.0 HDFS, but are not supported:

The following Apache Components are shipped as part of PHD 3.0 YARN, but are not supported:

  • Fair Scheduler

  • MapReduce Uber AM

  • MapReduce Eclipse Plug-in

Behavioral Changes

In PHD 3.0, behavioral changes affect the following Hadoop components:

  • Oozie

    Previous releases of the PHD Oozie client loaded the server-side configuration file, where JAVA_HOME is specific. The current version of the client does not load that server-side file. The workaround requires the following changes in its environmental setup:

    • The Oozie system environment requires JDK 1.6 or greater

    • The machine where the users will run the Oozie command line requires $JAVA_HOME to be set to version 1.6 or greater.

Tech Previews in This Release

All PHD 3.0 components listed here are official Apache releases of the most recent stable versions available. Pivotal’ philosophy is to provide patches only when absolutely necessary to assure the interoperability of the components. Unless you are explicitly directed by Pivotal Support to take a patch update, each of the PHD 3.0 components needs to remain at the following package version levels to ensure a certified and supported copy of PHD 3.0.

  • HDFS Encryption

  • YARN support for Docker

Apache Patch Information

This section describes Apache JIRAs that were addressed in PHD 3.0.

Hadoop Common/HDFS 2.6.0

PHD 3.0 provides Apache Hadoop Core 2.6.0 and the following additional Apache patches:

  • HDFS-3107. Introduce truncate.

  • HDFS-7009. Active NN and standby NN have different live nodes.

  • HDFS-7056. Snapshot support for truncate.

  • HDFS-7058. Tests for truncate CLI

  • HDFS-7263. Snapshot read can reveal future bytes for appended files.

  • HDFS-7425. NameNode block deletion logging uses incorrect appender.

  • HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume

  • HDFS-7470. SecondaryNameNode need twice memory when calling reloadFromImageFile.

  • HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes.

  • HDFS-7503. Namenode restart after large deletions can cause slow processReport

  • HDFS-7606. Fix potential NPE in INodeFile.getBlocks().

  • HDFS-7634. Disallow truncation of Lazy persist files.

  • HDFS-7638. Small fix and few refinements for FSN#truncate.

  • HDFS-7643. Test case to ensure lazy persist files cannot be truncated.

  • HDFS-7655. Expose truncate API for Web HDFS.

  • HDFS-7659. Truncate should check negative value of the new length

  • HDFS-7676. Fix TestFileTruncate to avoid bug of HDFS-7611.

  • HDFS-7677. DistributedFileSystem#truncate should resolve symlinks.

  • HDFS-7707. Edit log corruption due to delayed block removal again.

  • HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.

  • HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure.

  • HDFS-7738. Revise the exception message for recover lease; add more truncate tests such as truncate with HA setup, negative tests, truncate with other operations and multiple truncates.

  • HDFS-7760. Document truncate for WebHDFS.

  • HDFS-7831. Fix the starting index and end condition of the loop in FileDiffList.findEarlierSnapshotBlocks().

  • HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade.

  • HDFS-7885. Datanode should not trust the generation stamp provided by client.

  • HADOOP-926. Do not fail job history iteration when encounting missing directories.

  • HADOOP-941: Addendum patch.

  • HADOOP-11321. copyToLocal cannot save a file to an SMB share unless the user has Full Control permissions.

  • HADOOP-11368. Fix SSLFactory truststore reloader thread leak in KMSClientProvider.

  • HADOOP-11381. Fix findbugs warnings in hadoop-distcp, hadoop-aws, hadoop-azure, and hadoop-openstack

  • HADOOP-11381. Revert. (Fix findbugs warnings in hadoop-distcp, hadoop-aws, hadoop-azure, and hadoop-openstack.)

  • HADOOP-11412. POMs mention "The Apache Software License" rather than "Apache License".

  • HADOOP-11412. Revert. (POMs mention "The Apache Software License" rather than "Apache License"

  • HADOOP-11490. Expose truncate API via FileSystem and shell command.

  • HADOOP-11509. change parsing sequence in GenericOptionsParser to parse -D parameters before -files.

  • HADOOP-11510. Expose truncate API via FileContext.

  • HADOOP-11523. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB.

  • HADOOP-11579. Documentation for truncate.

  • HADOOP-11595. Add default implementation for AbstractFileSystem#truncate.

  • MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken service name properly.

  • YARN-570. Time strings are formated in different timezone.

  • YARN-2246. Made the proxy tracking URL always be http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections.

  • YARN-2571. RM to support YARN registry

  • YARN-2683. registry config options: document and move to core-default

  • YARN-2837. Support TimeLine server to recover delegation token when restarting.

  • YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher

  • YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie).

  • YARN-3103. AMRMClientImpl does not update AMRM token properly.

  • YARN-3207. Secondary filter matches entites which do not have the key being filtered for.

  • YARN-3227. Timeline renew delegation token fails when RM user's TGT is expired.

  • YARN-3239. WebAppProxy does not support a final tracking url which has query fragments and params.

  • YARN-3251. Fixed a deadlock in CapacityScheduler when computing absoluteMaxAvailableCapacity in LeafQueue.

  • YARN-3269. Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path.

HBase 0.98.4

PHD 3.0 provides HBase 0.98.4 and the following addtional Apache patches:

  • HBASE-212. Addendum. Fixes a unit test

  • HBASE-10499 In write-heavy scenario one of the regions does not get flushed causing RegionTooBusyException

  • HBASE-11569 Addendum for not skipping replayed edits for primary region replica

  • HBASE-12238 A few exceptions on startup - PARTIAL BACKPORT

  • HBASE-12533 staging directories are not deleted after secure bulk load

  • HBASE-12575 Sanity check table coprocessor classes are loadable

  • HBASE-12536. Reduce the effective scope of GLOBAL CREATE and ADMIN permission

  • HBASE-12562 Handling memory pressure for secondary region replicas - ADDENDUM for fixing findbug reported issues

  • HBASE-12791 HBase does not attempt to clean up an aborted split when the regionserver shutting down

  • HBASE-12714 RegionReplicaReplicationEndpoint should not set the RPC Codec

  • HBASE-12958 SSH doing hbase:meta get but hbase:meta not assigned

  • HBASE-13120 Allow disabling hadoop classpath and native library lookup

Hive 0.14.0

PHD 3.0 provides Hive 0.14.0 and the following additional Apache patches:

  • HIVE-480: HDFSCleanup thread holds reference to FileSystem

  • HIVE-6468: HiveServer2 (tcp mode, with SASL layer) OOMs when getting a non-thrift message

  • HIVE-6679: HiveServer2 should support TCP Keepalive & Server Socket Timeout on blocking calls

  • HIVE-7175: Provide password file option to beeline

  • HIVE-7270: Serde info is not shown in show create table statement, but shows in the desc table

  • HIVE-8295: Add batch retrieve partition objects for metastore direct sql

  • HIVE-8485: SUMMARY-Hive metastore NPE with Oracle DB when there is empty value for string for tblproperties/serdeproperties/etc, table not usable after creation

  • HIVE-8762 : HiveMetaStore.BooleanPointer should be replaced with an AtomicBoolean

  • HIVE-8791 : Hive permission inheritance throws exception S3

  • HIVE-8850: SUMMARY-[ObjectStore:: rollbackTransaction() needs to be looked into further

  • HIVE-8881: Receiving json "could not find job" error when web client tries to fetch all jobs from WebHCat but HDFS does not have the data.

  • HIVE-8888: Hive on Tez query output duplicate rows when there is explode in subqueries for joins

  • HIVE-8891: SUMMARY-[Another possible cause to NucleusObjectNotFoundException from drops/rollback

  • HIVE-8893 : Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode

  • HIVE-8966 : Delta files created by hive hcatalog streaming cannot be compacted

  • HIVE-9025 : join38.q (without map join) produces incorrect result when testing with multiple reducers

  • HIVE-9038: Join tests fail on Tez

  • HIVE-9055: Tez: union all followed by group by followed by another union all gives error

  • HIVE-9106 : improve the performance of null scan optimizer when several table scans share a physical path

  • HIVE-9112 : Query may generate different results depending on the number of reducers

  • HIVE-9141: HiveOnTez: mix of union all, distinct, group by generates error

  • HIVE-9155: HIVE_LOCKS uses int instead of bigint hive-txn-schema-0.14.0.mssql.sql

  • HIVE-9205/MHIVE-508: Change default Tez install directory to use /tmp instead of /user and create the directory if it does not exist

  • HIVE-9234: HiveServer2 leaks FileSystem objects in FileSystem.CACHE

  • HIVE-9235: Turn off Parquet Vectorization until all data types work: DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR

  • HIVE-9249: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables

  • HIVE-9278: Cached expression feature broken in one case

  • HIVE-9351: Running Hive Jobs with Tez cause templeton to never report percent complete

  • HIVE-9359: SUMMARY-[Hive export OOM error when table is huge. (32TB data, 4800+ partitions)]

  • HIVE-9382 : Query got rerun with Global Limit optimization on and Fetch optimization off

  • HIVE-9390: Enhance retry logic wrt DB access in TxnHandler

  • HIVE-9401: SimpleFetchOptimizer for limited fetches without filters

  • HIVE-9404 NPE in org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct()

  • HIVE-9436: SUMMARY-[RetryingMetaStoreClient does not retry JDOExceptions]

  • HIVE-9446: JDBC DatabaseMetadata.getColumns() does not work for temporary tables

  • HIVE-9473 : sql std auth should disallow built-in udfs that allow any java methods to be called

  • HIVE-9593. ORC reader should ignore new/unknown metadata streams.

  • HIVE-9652: STDERR redirection should not use in place updates TEZ UI

  • HIVE-9665: Parallel move task optimization causes race condition

  • HIVE-9673 : Set operationhandle in ATS entities for lookups

  • HIVE-9683: Client TCP keep-alive for Thrift as a workaround for THRIFT-2788

  • HIVE-9684: Incorrect disk range computation in ORC because of optional stream kind

  • HIVE-9743 Incorrect result set for vectorized left outer join

  • HIVE-9779 : ATSHook does not log the end user if doAs=false (it logs the hs2 server user)

  • HIVE-9836: Hive on Tez: fails when virtual columns are present in the join conditions (for e.g. partition columns)

  • HIVE-9841: IOException thrown by ORC should include the path of processing file

  • HIVE-9886: Tez NPE error when initialize reducer from 2 row_number function on each join side

  • HIVE-9892: HIVE schema failed to upgrade with schematool

  • HIVE-9832: Merge join followed by union and a map join in hive on tez fails.

  • MHIVE-479: Add input reader option to webhcat mapreduce streaming API

  • MHIVE-493 : change hadoop distcp command to perform directory copy and catch error logs if any

  • MHIVE-493 : Minor change to use destination file system while setting file status

  • MHIVE-493 : Use destination file system to get the destination file status

  • MHIVE-493 : Support MoveTask.moveFile to work with different file systems

  • MHIVE-527: add support to load and export between different filesystems

  • MHIVE-530: Removing hive.log.dir and the sortmerge noconditional task parameters from winpkg

  • MHIVE-542: Add hcatalog shared jars to the classpath

  • MHIVE-530: Remove unwanted properties from hive-site.xml of windows

  • PROD-559 The value of hive.zookeeper.quorum is not set in hive-site.xml

Knox 0.5.0

PHD 3.0 provides Knox 0.5.0 and the following additional Apache patches:

  • KNOX-492: Support service level replayBufferLimit for Ozzie, Hive and HBase.

Oozie 4.1.0

PHD 3.0 provides Apache Oozie 4.1.0 and the following additional Apache patches:

  • OOZIE-208. Adding missing oozie property oozie.service.HadoopAccessorService.hadoop.configurations to oozie install script.

  • PROD-594. Adding missing paranthesis for Function UpdateConfigWithAdditionalProperties in oozie install script.

Pig 0.14.0

PHD 3.0 provides Apache Pig 0.14.0 and the following additional Apache patches:

  • PIG-156: Pig command fails because the input line is too long

  • PIG-4334: PigProcessor does not set pig.datetime.default.tz

  • PIG-4342: Pig 0.14 cannot identify the uppercase of DECLARE and DEFAULT

  • PIG-4377: Skewed outer join produce wrong result in some cases (PIG-4377-2.patch)

  • PIG-4377: Skewed outer join produce wrong result in some cases

Ranger 0.4.0

PHD 3.0 provides Ranger 0.4.0 and the following additional Apache patch:

  • RANGER-188: Added LSB headers to Ranger Admin/Usersync init.d scripts

Tez 0.5.2

PHD 3.0 provides Apache Tez 0.5.2 and the following Apache patches:

  • TEZ-1642. TestAMRecovery sometimes fails.

  • TEZ-1775. Allow setting log level per logger.

  • TEZ-1800. Integer overflow in ExternalSorter.getInitialMemoryRequirement()

  • TEZ-1836. Provide better error messages when tez.runtime.io.sort.mb, spill percentage is incorrectly configured.

  • TEZ-1851. FileSystem counters do not differentiate between different FileSystems

  • TEZ-1852. Get examples to work in Local Mode.

  • TEZ-1861. Fix failing test: TestOnFileSortedOutput.

  • TEZ-1878. Task-specific log level override not working in certain conditions

  • TEZ-1924. Tez AM does not register with AM with full FQDN causing jobs to fail in some environments.

  • TEZ-1931. Publish tez version info to Timeline.

  • TEZ-1934. TestAMRecovery may fail due to the execution order is not determined.

  • TEZ-1942. Number of tasks show in Tez UI with auto-reduce parallelism is misleading.

  • TEZ-1949: Whitelist TEZ_RUNTIME_OPTIMIZE_SHARED_FETCH for broadcast edges

  • TEZ-1962. Fix a thread leak in LocalMode.

  • TEZ-1990. Tez UI: DAG details page shows Nan for end time when a DAG is running.

  • TEZ-2024. Compiliation error due to conflict.

  • TEZ-2024. TaskFinishedEvent may not be logged in recovery.

  • TEZ-2031. Tez UI: horizontal scrollbars do not appear in tables, causing them to look truncated.

  • TEZ-2037. Should log TaskAttemptFinishedEvent if taskattempt is recovered to KILLED

  • TEZ-2038. TEZ-UI DAG is always running in tez-ui when the app is failed but no DAGFinishedEvent is logged.

  • TEZ-2043. Tez UI: add progress info from am webservice to dag and vertex views.

  • TEZ-2052. Tez UI: log view fixes, show version from build, better handling of ats url config.

  • TEZ-2056. Tez UI: fix VertexID filter,show only tez configs by default,fix appattemptid.

  • TEZ-2063. Tez UI: Flaky log url in tasks table.

  • TEZ-2065. Setting up tez.tez-ui.history-url.base with a trailing slash can result in failures to redirect correctly.

  • TEZ-2068. Tez UI: Dag view should use full window height, disable webuiservice in localmode.

  • TEZ-2069. Tez UI: appId should link to application in dag details view.

  • TEZ-2077. Tez UI: No diagnostics on Task Attempt Details page if task attempt failed

  • TEZ-2078. Tez UI: Task logs url use in-progress url causing various errors.

  • TEZ-2079. Tez UI: trailing slash in timelineBaseUrl in ui should be handled.

  • TEZ-2092. Tez UI history url handler injects spurious trailing slash.

  • TEZ-2098. Tez UI: Dag details should be the default page for dag, fix invalid time entries for failed Vertices.

  • TEZ-2101. Tez UI: Issues on displaying a table.

  • TEZ-2102. Tez UI: DAG view has hidden edges, dragging DAG by holding vertex causes unintended click.

  • TEZ-2106. TEZ UI: Display data load time, and add a refresh button for items that can be refreshed.

  • TEZ-2112. Tez UI: fix offset calculation, add home button to breadcrumbs.

  • TEZ-2114. Tez UI: task/task attempt status is not available when its running.

  • TEZ-2116. Tez UI: dags page filter does not work if more than one filter is specified.

  • TEZ-2134. TEZ UI: On request failure, display request URL and server name in error bar.

  • TEZ-2135. ACL checks handled incorrectly in AMWebController.

  • TEZ-2136. Some enhancements to the new Tez UI.

  • TEZ-2142. TEZ UI: Breadcrumb border color looks out of place in wrapped mode.

  • TEZ-2158. TEZ UI: Display dag/vertex names, and task/attempt index in breadcrumb

  • TEZ-2160. Tez UI: App tracking URL should support navigation back.

  • TEZ-2165. Tez UI: DAG shows running status if killed by RM in some cases.

ZooKeeper 3.4.6

PHD 3.0 provides ZooKeeper 3.4.6 and the following additional Apache patches:

  • ZOOK-46: Port ZooKeeper-1506 that handles connection issues by retrying DNS

Fixed in PHD 3.0

The following features and fixes were contributed back to Apache with the release of PHD 3.0:

Pivotal Bug No.

Apache JIRA

Description

Component

BUG-28115

HADOOP-11321

CopyToLocal cannot save a file to an SMB share unless the user has full Control permissions

Hadoop Common

BUG-32505

HADOOP-11368

Fix SSLFactory truststore reloader thread leak in KMSClientProvider

HDFS

BUG-28620

HADOOP-9629

Synchronize WASB code from upstream Apache

Hadoop Common

BUG-33034

HBASE-12238

Exceptions on startup

HBase

BUG-26389

HBASE-12575

Botched alter table involving coprocessor crashes RS

HBase

BUG-29996

HBASE-12791

HBase does not attempt to clean up an aborted SPLIT when the Regions Server is shutting down

HBase

BUG-31373

HBASE-12958

SSH doing hbase:meta displays "hbase:meta not assigned" message

HBase

BUG-32330

HBASE-13120

Allow native library path to be overriden

HBase

BUG-32106

HDFS-7009

Active NN and standby NN have different live nodes

HDFS

BUG-28206

HDFS-7425

NameNode block deletion logging uses incorrect appender.

HDFS

BUG-29862

HDFS-7489

Incorrect locking in FsVolumeList#checkDirs can hang datanodes.

HDFS

BUG-29478

HDFS-7560

ACLs removed by removeDefaultAcl() will be back after NameNode restart/failover

HDFS

BUG-31221

HDFS-7707

Edit log corruption due to delayed block removal again

HDFS

BUG-31411

HDFS-7714

Simultaneous restart of HA NameNodes and DataNode can causes DataNode to register successfully with only one NameNode.

HDFS

BUG-31164

HDFS-7733

NFS: readdir/readdirplus returns null directory attribute on failure

HDFS

BUG-32197

HDFS-7843

Truncate test failed during rollback

HDFS

BUG-33108

HDFS-7925

HDFS truncate command fails

HDFS

BUG-30864

HIVE-5700, HIVE-9445

Revert HIVE-5700 - enforce single date format for partition column storage

Hive

BUG-29572

HIVE-7270

Serde info is not shown in SHOW CREATE TABLE statement, but shows in the desc table

Hive

BUG-27598

HIVE-8295

DirectSQL with Oracle metastore fails with error ORA-01795, Maximum number of expression can not be more than 1000

Hive

BUG-29164

HIVE-8870

Errors when selecting a struct field within an array from ORC based tables

Hive

BUG-27856

HIVE-8881

Receiving json error when web client tries to fetch all jobs from webhcat where HDFS does not have the data.

WebHCat

BUG-28232

HIVE-8888

Hive on Tez query output duplicate rows when JOIN subqueries are expanded

Hive

BUG-30663

HIVE-8893, HIVE-9473

UDF that allows untrusted code execution should be disabled

Hive

BUG-26804

HIVE-8948

Fix or disable flaky test testTransactionBatchEmptyCommit

Hive

BUG-29631

HIVE-8966

Delta files created by hive hcatalog streaming cannot be compacted

Hive

BUG-30288

HIVE-9025, HIVE-9112

Incorrect result due to bug in constant propagation optimizer

Hive

BUG-29161

HIVE-9051

TezJobMonitor in-place updates logs too often to logfile

Hive

BUG-29158

HIVE-9067

OrcFileMergeOperator may create merge file that does not match properties of input files

Hive

BUG-29162

HIVE-9090

Rename "Tez File Merge Work" to smaller name

Hive

BUG-29159

HIVE-9111

Potential NPE in OrcStruct for list and map types

Hive

BUG-29840

HIVE-9249

java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables

Hive

BUG-30063

HIVE-9278

Cached expression feature broken in one case

Hive

BUG-30305

HIVE-9359

Hive export OOM error when table is huge. (32TB data, 4800+ partitions)

Hive

BUG-31213

HIVE-9593

ORC reader should ignore new/unknown metadata streams

Hive

BUG-32081

HIVE-9652

STDERR redirection should not use in-place updates TEZ UI

Hive

BUG-31288

HIVE-9665

ParallelMoveTask test failed on Ambari-deployed cluster due to OOM error

Hive

BUG-32089

HIVE-9743

Incorrect result set for left outer join when executed with tez versus mapreduce

Hive

BUG-32751

HIVE-9841

IOException thrown by ORC should include the path of processing file

Hive

BUG-29423

HADOOP-799

NullPointerException in FSDataset

MapReduce

BUG-28571

HBASE-12533

Staging directories are not deleted after secure bulk load

HBase

BUG-29367

HDFS-7443

Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume

HDFS

BUG-30397

HDFS-7470

SecondaryNameNode need twice memory when calling reloadFromImageFile

HDFS

BUG-27347

HDFS-7503

Namenode restart after a large deletion could cause slow processReport.

HDFS

BUG-28815

HIVE-6679

HiveServer2 should support TCP Keepalive & Server Socket Timeout on blocking calls

Hive

BUG-32184

HIVE-9779

ATSHook does not log the end user if doAs=false (it logs the hs2 server user)

Hive

BUG-32273

n/a

Unable to change Hue users' passwords via the Hue UI

Hue

BUG-32998

n/a

Fix error on deleting users when upgrading Hue

Hue

BUG-32389

n/a

Users cannot use the Job Browser to investigate failed jobs

Hue

BUG-31139

n/a

Fix django-db.utils.DatabaseError: current transaction is aborted when setting up Hue on PostgreSQL

Hue

BUG-28353

KNOX-464

Knox not returning proper redirection url when executing file creation/upload operation via load balancer

Knox

BUG-31968

OOZIE-2119

Oozie Distcp action fails with error : Returned value from distcp is non-zero (-1)

Oozie

BUG-30095

RANGER-192

User Detail Page hangs if user has many groups

Ranger

BUG-32094

TEZ-2135

ACL checks working incorrectly in AMWebController

Tez

BUG-29254

YARN-2964

RM prematurely cancels tokens for jobs that submit jobs (oozie)

YARN

BUG-31944

YARN-3239

WebAppProxy in YARN broken for redirecting to the Tez View in Ambari

YARN

BUG-29055

YARN-570

Time strings are formated in different timezone

YARN

BUG-20353

YARN-570

Resource Manager Timezone only appears in GMT, should appear in Local Time to Browser

YARN

BUG-31195

KNOX-473

Knox fails to rewrite Load balancer URI when doing out request

Knox

BUG-30967

KNOX-491

Increase default replay buffer size to 8K

Knox

BUG-30968

KNOX-492

Support service level replayBufferLimit for Oozie, Hive and HBase

Knox

BUG-28636

PIG-4334

PigProcessor does not set pig.datetime.default.tz

Pig

BUG-31704

RANGER-242

Ranger on Ambari: Ranger config migration script fails to migrate ranger-admin/install.properties configs to Ranger config properties under Ambari.

Ranger

BUG-29175

TEZ-1775

Allow setting log level per logger (See also TEZ-1878)

Tez

BUG-29373

TEZ-1851

FileSystem counters do not differentiate between different FileSystems

Tez

BUG-31540

TEZ-2024, TEZ-2037, TEZ-2087

Recovered Task/TaskAttempt status is not available on tez-ui

Tez

BUG-31640

TEZ-2029

Issues on DAG's view

Tez

BUG-31481

TEZ-2038

DAG is always in running status in tez-ui when the app is failed before DAG is completed

Tez

BUG-31626

TEZ-2050, TEZ-2114

No Running Status shows for Task/TaskAttempt

Tez

BUG-31637

TEZ-2094, YARN-3207

Running Task/TaskAttempt are not excluded when choosing other status in the Dropdownlist

Tez, YARN

BUG-27488

YARN-2837

Timeline server needs to recover the timeline DT when restarting

YARN

BUG-32619

YARN-3227

HiveServer2 long running test is failing due to TezSession not being able to renew TIMELINE_DELEGATION_TOKEN

YARN

Known Issues for Pivotal HD 3.0

Key

Apache JIRA

Summary

Component/s

RMP-2381

 

Sub-5-Second: Lookup Query Metastore in less than 200 milliseconds (tech preview)

Hive

RMP-2377

HIVE-7926

Sub-5-Second: LLAP Daemons and LLAP API (tech preview)

Hive

RMP-3332

HIVE-9792

RMP-2430 SQL: Partition pruning support for Intervals

Hive

RMP-3333

HIVE-9792

RMP-2430 SQL: ORC PPD support for Intervals

Hive

RMP-2448

 

Sub-5-Second: Umbrella: Execute TPC-DS query 55 in under 5 second at 30TB and Query 27 at 1 TB (tech preview)

Hive

RMP-2662

 

Sub-Second: Vectorization enhancements - MapJoin

Hive

RMP-2620

 

CBO: Introduce semi join reduction in Calcite

Hive

RMP-2449

HIVE-9675

Transactions: Support BEGIN, COMMIT, ROLLBACK

Hive

RMP-2634

 

Replication for Hive (partial)

Hive

RMP-2430

HIVE-9792

SQL: Implement Interval types and datetime arithmetic (partial)

Hive

BUG-26944

 

E0803: IO error, null during scheduling feed on a cluster with wire enc on

Oozie

BUG-33113

 

HBase-HA master fail over immediately

HBase

BUG-31943

 

test_IntegrationTestRegionReplicas region not serving

HBase

BUG-32509

 

test_runIntegrationTestMTTRwithDistributedLogs due to failure in testKillRsHoldingMeta

HBase

BUG-31353

 

IntegrationTestRegionReplicaPerf region not serving

HBase

BUG-29903

 

IntegrationTestMTTR slowness

HBase

BUG-31936

 

test_hbaseBigLinkedList[Loop] TimeoutException

HBase

BUG-27858

 

HBase HA tests fail with Table Namespace Manager not ready yet

HBase

BUG-27509

HBASE-12464

test_ReplicaCopyTable_verifyCopyTable[tableone] fails after trying several times for 'Failed to find location, tableName=hbase:meta, row='

HBase

BUG-27256

 

Create mr based Hbase Long running application for Rollingupgrade

HBase

BUG-25961

 

HBase tests failing with org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out; 10000ms

HBase

BUG-23619

 

On Windows, tests fail complaining about Cannot run program /usr/bin/env

HBase

BUG-23981

 

IOException: Invalid HFile block magic in Windows unit tests

HBase

BUG-22841

 

HBase test failed with Insufficient permissions (user=gpadmin@example.com error

HBase

BUG-21254

 

hbck tool fixMeta flag not able to recreate deleted entries from hbase:meta table when just one column is deleted

HBase

BUG-23897

 

HBase REST server continuously restarting in secure windows env

HBase

BUG-33278

HBASE-13239

HBASE grants at specific column level does not work for Groups

HBase

BUG-27656

HBASE-12472, HBASE-13192

IntegrationTestBulkLoad fails on mapreduce

HBase

BUG-27756

 

test_IntegrationTestMultiGetRegionReplicas slowness

HBase

BUG-31822

 

hcat startup failed on windows

HCatalog

BUG-23590

HIVE-8247

Pig cursor written to Hive via HCat doesn't NULL-fill missing columns

HCatalog

BUG-27971

HDFS-7558

HDFS daemons JVM pauses when the NFS client and server on the same host as other HDFS daemons

HDFS

BUG-25352

 

Exit code 0 when command (put/copying a file bigger than quota) fails.

HDFS

BUG-31624

 

Dropping a 2000+ partition table takes over 10 minutes

Hive

BUG-32416

 

Hive result mismatch in decimal digits when converting from double

Hive

BUG-31671

 

HS2 concurr tests intermittently fail with http mode on secure Openstack throwing org.apache.hive.service.auth.HttpAuthenticationException: Kerberos authentication failed

Hive

BUG-28097

HIVE-8915

Log file explosion due to non-existence of COMPACTION_QUEUE table

Hive

BUG-30984

 

Hive compaction tests failing

Hive

BUG-33167

 

Hive does not prevent non whitelisted config params from being set under certain condtions

Hive

BUG-32795

 

HIVE schema failed to upgrade with schematool

Hive

BUG-33168

 

hive authorization checks on configs should not be done if authorization is diabled

Hive

BUG-28218

 

Hive CLI hangs

Hive

BUG-27636

 

Oracle: intemittent acid_concurrency test failures due to NoSuchLockException

Hive

BUG-26243

 

CBO : Q51 fails in explain with Failed to breakup Windowing invocations into Groups.

Hive

BUG-27507

 

Hive CLI returns 0 even in case of failures

Hive

BUG-27592

 

Beeline Client Connections Errors - Obscure errors and exception swallow - 4

Hive

BUG-29427

 

Turn off Parquet Vectorization until all data types work: DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR

Hive

BUG-27582

 

Beeline Client Connections Errors - Obscure errors and exception swallow - 1

Hive

BUG-30022

HIVE-7693

Invalid column ref error in order by when using column alias in select clause and using having

Hive

BUG-25058

HIVE-8459

DbLockManager reader locks are locking tables in addition to partitions

Hive

BUG-25064

 

Hive : Q51 fails in CBO with "Failed to breakup Windowing invocations into Groups. At least 1 group must only depend on input columns"

Hive

BUG-23260

 

BUG-22878 insert after alter table issue

Hive

BUG-23830

HIVE-8263

CBO : Query query has date_dim d1,date_dim d2 and date_dim d3 but the explain has d1, d1 and d1

Hive

BUG-21468

 

With CBO enabled queries fail with Failed to breakup Windowing invocations into Groups

Hive

BUG-20856

 

Column names are missing from join expression in Map join with CBO enabled

Hive

BUG-18450

HIVE-7160

Vectorization Udf: GenericUDFConcat, is not supported for String+Long

Hive

BUG-23874

 

Map join sizing done by auto.convert.join.noconditionaltask.size doesn't take into account Hash table overhead and results in OOM

Hive

BUG-25424

 

When table is renamed stats are lost as changes are not propagated to metastore tables TAB_COL_STATS and PART_COL_STATS

Hive

BUG-27637

 

Postgres: intermittent test failure due to "ERROR: could not serialize access due to read/write dependencies among transactions"

Hive

BUG-33338

 

Hiveserver2 in HTTP mode is not applying auth_to_local rules

Hive

BUG-32859

HIVE-9977

Hive metastore auto major compaction not happening if data is loaded through dynamic partitioning insert

Hive

BUG-33321

 

Created tables not available under Hcatalog

Hue

BUG-28427

 

NPE in ambari knox install

Knox

BUG-27094

 

Knox HBase tests skipped for ambari test config

Knox

BUG-27091

 

Knox YarnSysTest skipped in secure mode for ambari test config

Knox

BUG-27089

 

Knox YarnSysTest skipped in secure mode for gsInstaller test config

Knox

BUG-28039

 

Knox Failed Authentication Auditing

Knox

BUG-28031

 

Knox rolling upgrade system test support for wire encryption

Knox

BUG-27802

 

Improve reliability of Knox smoke test in system test deployment

Knox

BUG-27906

 

Increase timeout of system test WebHCatDslSysTest-testPig

Knox

BUG-27549

 

Rolling upgrade: knox - webHbase smoke test failed after upgrading core components

Knox

BUG-25049

 

Two WebHdfsHaFuncTest are unstable and should be fixed and enabled

Knox

BUG-23889

 

Windows security scripting should setup Knox to use ActiveDirectory for LDAP

Knox

BUG-32419

 

TestBundleVersion: oozie bundle takes 1 hr 10mins to recognized a three completed/killed coordinator/wf actions

Oozie

BUG-32763

 

yum cannot find Xvfb for centos06

Oozie, RelEng

BUG-32690

 

test_pig_via_oozie_ha[resourcemanager] exception:NotFoundException,message:java.lang.Exception: job, job_1425457888074_0001, is not found,javaClassName

Oozie

BUG-31426

 

Oozie failed to start due to "SA" schema not created

Oozie, RelEng

BUG-32026

 

test_mapred_via_oozie_ha[oozie-nn-rm-5-jobs] FAILED

Oozie

BUG-31349

 

oozie job in running state is causing intermittent timeout failures

Oozie

BUG-33180

 

oozie logging tests fail on MR framework with postgres 9

Oozie

BUG-26766

 

[Rolling Upgrade] Allow Prepare Oozie.war to be done without stopping Oozie Server

Oozie

BUG-28614

 

Oozie property oozie.authentication.kerberos.principal is being used for authentication of webhdfs endpoint

Oozie

BUG-24517

 

oozie needs to support Application Tag

Oozie

BUG-23655

 

Oozie stuck on mapreduce job because workflow state isn't changing from RUNNING to KILLED

Oozie

BUG-24118

 

PHD Windows: Update/clean up oozie's SQL Server JDBC driver copies

Oozie

BUG-23527

 

Oozie config referred to versioned hadoop config

Oozie

BUG-26428

 

XaAdmin and XaAgent jobs for Debian failing on Unix clusters.

Ranger

BUG-31949

 

In hive plugin TABLE policy restriction for all tables {{*}} fails when UDF policy for all functions {{*}} is maintained for a database.

Ranger

BUG-31749

 

Usersync NPE when object does not have userNameAttribute

Ranger

BUG-31782

 

Ranger gives error when using hyphen or space in lastname field

Ranger

BUG-33056

 

{start,stop}-thriftserver equivalent script on Windows is missing

Spark

BUG-33054

 

Spark examples are packaged differently on Windows

RelEng, Spark

BUG-33335

 

spark make-distribution.sh doesn't evaluate SPARK_HIVE on OSX

Spark

BUG-32342

 

Minimum/Maximum Duration show a empty bracket next to 0 secs when you purposefully failed a job.

Tez

BUG-31522

 

You cannot search by the name of 'Queue'

Tez

BUG-31526

 

The order of columns will go back to the initial setting after you refresh a page or come back from another page.

Tez

BUG-31598

 

Tez UI vertical scroll broken from time to time

Tez

BUG-31594

 

TEZ view UI Column selector should have a select all or something to make selecting columns easier

Tez

BUG-31565

 

When you kill a job in the middle of running it, it Diagnostics shows 'killed/failed due to:null".

Tez

BUG-26419

 

Tez application fails with diff client and AM version

Tez

BUG-27562

 

WordCount on Tez with non-nightly failed with NPE

Tez

BUG-31583

 

Tez UI : DAG fails but Task attempts don't show any failues

Tez

BUG-31602

 

Tez UI :Task Tab should have Vertex name as Hyper link not Vertex ID

Tez

BUG-31595

 

Tez UI : Per DAG counters not getting aggregated

Tez

BUG-31623

 

Tez UI : Increase the size of table name tex box in the DAG view

Tez

BUG-24831

 

TEZ windows test framework needs to respect ${fs.defaultFS} in tez configs.

Tez

BUG-25194

 

AM running out of memory for partitioned bucketed tables

Tez

BUG-25380

 

For hive to work with Virtual columns and parittion names, we need information available within a record-reader

Tez

BUG-24518

 

Tez needs to support Application Tag

Tez

BUG-29279

 

Distributedshell App fails after unregistering the application

YARN

BUG-27194

 

[YARN-2821] Failures encountered on Distributed Shell on windows

YARN

BUG-26907

HADOOP-11274

[HADOOP-11274] ConcurrentModificationException in Configuration Copy Constructor

YARN

BUG-27590

 

Distributed shell job does not execute python script in Windows

YARN

BUG-26875

 

Sometimes NodeManager keeps in tracking non-exitant completed container process

YARN

BUG-22522

 

MR Job failed when run via oozie and NN and RM were being killed

YARN

BUG-33546

YARN-3393

Getting application(s) goes wrong when app finishes before starting the attempt

YARN

BUG-27632

 

Add feature of retry in zookeeper client

Zookeeper

BUG-30784

ZOOKEEPER-1952

zookeeper.log.file property is not respected; log output goes only to the zookeeper.out

Zookeeper

BUG-19255

 

Uninstalling the ZooKeeper RPM does not remove all ZooKeeper directories

Zookeeper