Steps to Reproduce:

1. Configure hue.ini as follows to use IAM role detection:
   [aws]
   [[aws_accounts]]
   [[[default]]]
   has_iam_detection=true

2. Annotate the Kubernetes service account with the necessary roleArn for AWS IAM role usage.

3. Confirm the role has s3:PutObject, s3:GetObject, and other necessary S3 permissions. (I verified this by successfully using the role for similar S3 operations in another application.)

4. Open the Hue file browser, navigate to S3, and attempt to upload a file.

Observed Behavior:

While read and list operations work as expected with has_iam_detection=true, the file upload fails with a 403 Forbidden error:

aws.s3.s3fs.S3FileSystemException: Failed to access path "": User is not authorized to access path "".

When I provide AWS access keys and secrets directly in the hue.ini configuration, the upload works without issue. This suggests a potential difference in how Boto (or another underlying component) is configured to handle IAM role-based access compared to direct access keys.

Additional Details:

Role Verification: The IAM role used has been validated to have the necessary permissions (including s3:PutObject) in another application, and write operations work fine there.
Environment: Hue is running within a Kubernetes cluster, with the service account annotated with the IAM role.
Boto Library: Hue appears to be using Boto 2.x for S3 operations. This could be relevant since Boto 2.x and Boto3 handle IAM role-based access differently.
Hue Version: 4.11

Possible Areas to Investigate:

Boto IAM Role Support: Is there a known limitation with Boto 2.x for IAM role usage in Hue? If so, are there any recommended workarounds?
Configuration Differences: Are there any additional configuration steps needed to ensure full IAM role compatibility for upload operations in Hue?

Expected Behavior:

When using has_iam_detection=true, Hue should detect and utilize the IAM role for all S3 operations, including file uploads, without requiring access keys in the configuration file.

Logs:

Here are the relevant log entries from an upload attempt:

[06/Nov/2024 00:48:02 -0800] decorators ERROR Error running guess_format
...
aws.s3.s3fs.S3FileSystemException: Failed to access path "s3a://myBucket/...": User is not authorized to access path at "s3a://myBucket/...".

Request:

Could the maintainers provide insights or guidance on whether this is a known issue or if there are specific configurations needed to fully support IAM roles with has_iam_detection=true for S3 uploads? If there are any recommended versions of Boto or specific patches for this use case, please advise.

The entire directory /opt/hue/hue-release-4.11.0/build doesn’t seem to get created. What mechanism is responsible for creating the virtual environment directories and how can this be resolved? I’ve tried manually creating the directory so the ‘stamp’ gets touched, but this results in errors being thrown in the resulting files.

I have been using Hue for a bit now, and I am really enjoying how it simplifies data querying and exploration. However, I have noticed that some of my queries are starting to take longer to execute, especially when working with larger datasets.

I have gone through these resources/articles Hue Performance Tuning Guide Mendix Platform Features and they are quite informative but seem too complicated so I want to learn from community.

I am wondering if anyone has tips or best practices for optimizing query performance in Hue? Are there specific configurations or settings I should be looking into? Also, if there are any tricks to better handle large datasets, I would love to hear them!

Appreciate any advice you can share. Thanks in advance!

Best Regards

Some times back I got the same issue but I not able to recall the process. Let me check it and get back to you soon till then you should try solution posted by @bjorn

Thanks

Follow this- Installation steps Hue

Thanks

Use this code-

SELECT
HOUR(date_) AS hour,
MINUTE(date_) AS minute,
SECOND(date_) AS second
FROM
your_table;

Try this and let me know if it’s working

Thanks

Our team has recently adopted Hue for data visualization and querying within our company’s extensive big data environment. While we’ve found the platform incredibly valuable, we’re encountering performance challenges as our data volume grows.

I’m reaching out to the community for expert advice on optimizing Hue for large-scale data processing. We’re particularly interested in:

• Configuration best practices: Are there specific Hue settings to enhance performance with massive datasets?
• Resource allocation: How can we effectively distribute resources (memory, CPU) for optimal Hue operation?
• Query optimization techniques: What strategies can we employ to efficiently handle large-scale data queries?
• Complementary tools: Are there any recommended integrations or tools to boost Hue’s performance?
• We’re utilizing Hue on a Hadoop ecosystem for complex queries and interactive visualizations. Any insights, documentation, or real-world examples would be immensely helpful.

I also check this: https://discourse.gethue.com/t/hue-pyspark-connector-using-livy-how-to-change-spark-driver-memorlooker But I have not found any solution. Could anyone provide me the best solution for this?

Thank you for sharing!

Respected community member

I think you should try to the Cloudera SQL AI Assistant integration with Hue is proprietary and while the assistant itself is not open source, the integration parts relevant to Hue are typically merged into the open source gethue repository. However some features or specific implementations related to Cloudera’s SQL AI Assistant may remain proprietary and not be included in the open-source codebase.

Thanks

If you want to date and time from the date column, you can use the built in function for the hour function; for the minute; and for the secound; imute use the secound function.
Try it and then let me know about the result

As per my knowledge, firstly you need to check Hue Version and then Inspect Logs, Browser Compatibility, Hue Configuration and at last Alternative Methods. I hope this will help you.

Respected community member

I will note I am running hue as root because when i try to run as a dedicated hue user I am similarly running into a permission error when hue is trying to run a os.setgid call, which non-root users shouldnt have permissions to do.

I’m looking into where I can tweak the permissions for these /tmp files, but if anyone notices any mistakes I’m making that would cause this problem please let me know. Thanks!

I have been trying to get Hue running on RHEL9 against python 3.9. I am currently running into the following error when starting the server with build/env/bin/supervisor:

Environment:

Request Method: GET
Request URL: http://localhost:8888/hue/accounts/login

Django Version: 3.2.25
Python Version: 3.9.18
Installed Applications:
[‘django.contrib.auth’,
‘django.contrib.contenttypes’,
‘django.contrib.sessions’,
‘django.contrib.sites’,
‘django.contrib.staticfiles’,
‘django_extensions’,
‘django_babel’,
‘desktop’,
‘axes’,
‘webpack_loader’,
‘django_prometheus’,
‘crequest’,
‘rest_framework’,
‘rest_framework.authtoken’,
‘drf_spectacular’,
‘drf_spectacular_sidecar’,
‘indexer’,
‘metadata’,
‘notebook’,
‘dashboard’,
‘kafka’,
‘about’,
‘beeswax’,
‘filebrowser’,
‘help’,
‘hive’,
‘jobbrowser’,
‘jobsub’,
‘metastore’,
‘oozie’,
‘proxy’,
‘spark’,
‘useradmin’,
‘zookeeper’,
‘corsheaders’]
Installed Middleware:
[‘corsheaders.middleware.CorsMiddleware’,
‘desktop.middleware.MetricsMiddleware’,
‘desktop.middleware.EnsureSafeMethodMiddleware’,
‘desktop.middleware.AuditLoggingMiddleware’,
‘desktop.middleware.MultipleProxyMiddleware’,
‘django.middleware.common.CommonMiddleware’,
‘django.contrib.sessions.middleware.SessionMiddleware’,
‘django.contrib.auth.middleware.AuthenticationMiddleware’,
‘desktop.middleware.ProxyMiddleware’,
‘desktop.middleware.SpnegoMiddleware’,
‘desktop.middleware.HueRemoteUserMiddleware’,
‘django.middleware.locale.LocaleMiddleware’,
‘django_babel.middleware.LocaleMiddleware’,
‘desktop.middleware.AjaxMiddleware’,
‘django.middleware.security.SecurityMiddleware’,
‘django.middleware.clickjacking.XFrameOptionsMiddleware’,
‘desktop.middleware.ContentSecurityPolicyMiddleware’,
‘desktop.middleware.LoginAndPermissionMiddleware’,
‘django.contrib.messages.middleware.MessageMiddleware’,
‘desktop.middleware.NotificationMiddleware’,
‘desktop.middleware.ExceptionMiddleware’,
‘desktop.middleware.ClusterMiddleware’,
‘django.middleware.csrf.CsrfViewMiddleware’,
‘desktop.middleware.CacheControlMiddleware’,
‘django.middleware.http.ConditionalGetMiddleware’,
‘desktop.middleware.MimeTypeJSFileFixStreamingMiddleware’,
‘crequest.middleware.CrequestMiddleware’,
‘desktop.middleware.EnsureSafeRedirectURLMiddleware’,
‘useradmin.middleware.LastActivityMiddleware’,
‘axes.middleware.AxesMiddleware’]

Traceback (most recent call last):
File “/usr/lib64/python3.9/multiprocessing/managers.py”, line 802, in _callmethod
conn = self._tls.connection

During handling of the above exception (‘ForkAwareLocal’ object has no attribute ‘connection’), another exception occurred:
File “/usr/gmdp/current/hue/build/env/lib/python3.9/site-packages/django/core/handlers/exception.py”, line 47, in inner
response = get_response(request)
File “/usr/gmdp/current/hue/build/env/lib/python3.9/site-packages/django/utils/deprecation.py”, line 116, in call
response = self.process_request(request)
File “/usr/gmdp/current/hue/desktop/core/src/desktop/middleware.py”, line 888, in process_request
global_registry().update_metrics_shared_data()
File “/usr/gmdp/current/hue/desktop/core/src/desktop/lib/metrics/registry.py”, line 128, in update_metrics_shared_data
self._metrics_dict[os.getpid()] = metrics
File “”, line 2, in setitem

File “/usr/lib64/python3.9/multiprocessing/managers.py”, line 806, in _callmethod
self._connect()
File “/usr/lib64/python3.9/multiprocessing/managers.py”, line 793, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File “/usr/lib64/python3.9/multiprocessing/connection.py”, line 506, in Client
c = SocketClient(address)
File “/usr/lib64/python3.9/multiprocessing/connection.py”, line 634, in SocketClient
s.connect(address)

Exception Type: PermissionError at /hue/accounts/login
Exception Value: [Errno 13] Permission denied

I’m using the /usr/gmdp/current/hue installation location because im trying to adapt an old mpack I found so the service can be managed by Ambari, i plan on installing under /usr/local/hue in the future. I’ve been stuck for a couple weeks with this one, so thought I’d post here. Any ideas?

Error Log

[root@cve1 hue]# /usr/lib/hue/build/env/bin/pip2.7 install mysqlclient
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Requirement already satisfied: mysqlclient in /usr/lib64/python2.7/site-packages (1.4.6)
[root@cve1 hue]# cd /usr/lib/hue/build/env
[root@cve1 env]# bin/hue syncdb
/usr/lib/hue/build/env/lib/python2.7/site-packages/requests_kerberos-0.12.0-py2.7.egg/requests_kerberos/kerberos_.py:11: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography import x509
[18/Apr/2024 14:52:54 +0000] settings     DEBUG    DESKTOP_DB_TEST_NAME SET: /usr/lib/hue/desktop/desktop-test.db
[18/Apr/2024 14:52:54 +0000] settings     DEBUG    DESKTOP_DB_TEST_USER SET: hue_test
Traceback (most recent call last):
  File "bin/hue", line 14, in <module>
    load_entry_point('desktop', 'console_scripts', 'hue')()
  File "/usr/lib/hue/desktop/core/src/desktop/manage_entry.py", line 239, in entry
    raise e
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: libmysqlclient.so.18: cannot open shared object file: No such file or directory.
Did you install mysqlclient or MySQL-python?
[root@cve1 env]# /usr/lib/hue/build/env/bin/pip install mysql-python
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Requirement already satisfied: mysql-python in ./lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg (1.2.5)

As i understand Cloudera SQL AI Assistant is integrated with Hue (https://blog.cloudera.com/setting-up-and-getting-started-with-clouderas-new-sql-ai-assistant/).
Is support merged to opensource gethue repository?
Probably assistant itself is proprietary, but integration part is in gethue?