Commit Graph

678 Commits

Author SHA1 Message Date
Anis Eleuch 69c9496c71
Upgrade github.com/minio/pkg/v2 and other deps (#19747) 2024-05-15 11:04:40 -07:00
Klaus Post c36eaedb93
Re-add "Fix incorrect merging of slash-suffixed objects (#19729)
Adds regression test for #19699

Failures are a bit luck based, since it requires objects to be placed on different sets.

However this generates a failure prior to #19699

* Revert "Revert "Fix incorrect merging of slash-suffixed objects (#19699)""

This reverts commit f30417d9a8.

* Don't override when suffix doesn't match. Instead rely on quorum for each.
2024-05-13 09:30:24 -07:00
Andreas Auernhammer 8b660e18f2
kms: add support for MinKMS and remove some unused/broken code (#19368)
This commit adds support for MinKMS. Now, there are three KMS
implementations in `internal/kms`: Builtin, MinIO KES and MinIO KMS.

Adding another KMS integration required some cleanup. In particular:
 - Various KMS APIs that haven't been and are not used have been
   removed. A lot of the code was broken anyway.
 - Metrics are now monitored by the `kms.KMS` itself. For basic
   metrics this is simpler than collecting metrics for external
   servers. In particular, each KES server returns its own metrics
   and no cluster-level view.
 - The builtin KMS now uses the same en/decryption implemented by
   MinKMS and KES. It still supports decryption of the previous
   ciphertext format. It's backwards compatible.
 - Data encryption keys now include a master key version since MinKMS
   supports multiple versions (~4 billion in total and 10000 concurrent)
   per key name.

Signed-off-by: Andreas Auernhammer <github@aead.dev>
2024-05-07 16:55:37 -07:00
Alex 1e83f15e2f
Update Console version to v1.4.0 (#19684) 2024-05-06 19:47:37 -07:00
Harshavardhana 08ff702434
enhance ListSVCs() API to return more info to avoid InfoSvc() (#19642)
ConsoleUI like applications rely on combination of

ListServiceAccounts() and InfoServiceAccount() to populate
UI elements, however individually these calls can be slow
causing the entire UI to load sluggishly.
2024-05-01 05:41:13 -07:00
Harshavardhana f4f1c42cba
deprecate usage of sha256-simd (#19621)
go1.21 already implements the necessary optimizations
2024-04-25 23:31:35 -07:00
Cesar N 943d815783
Update Console UI to v1.3.0 (#19617) 2024-04-25 12:12:03 -07:00
Harshavardhana d95e054282
update all deps regular cadence (#19523) 2024-04-16 11:48:56 -07:00
Allan Roger Reid 7c1f9667d1
Use GetDuration() helper for MINIO_KMS_KEY_CACHE_INTERVAL as time.Duration (#19512)
Bonus: Use default duration of 10 seconds if invalid input < time.Second is specified
2024-04-16 08:43:39 -07:00
Alex 8d7d4adb91
Updated Console UI to v1.2.0 (#19467) 2024-04-11 01:31:16 -07:00
Harshavardhana 1bb670ecba
use new generics based LRU from hashicorp (#19409)
we have been using an LRU caching for internode
auth tokens, migrate to using a typed implementation
and also do not cache auth tokens when its an error.
2024-04-04 11:58:48 -07:00
Aditya Manthramurthy c9e9a8e2b9
fix: ldap: use validated base DNs (#19406)
This fixes a regression from #19358 which prevents policy mappings
created in the latest release from being displayed in policy entity
listing APIs.

This is due to the possibility that the base DNs in the LDAP config are
not in a normalized form and #19358 introduced normalized of mapping
keys (user DNs and group DNs). When listing, we check if the policy
mappings are on entities that parse as valid DNs that are descendants of
the base DNs in the config.

Test added that demonstrates a failure without this fix.
2024-04-04 11:36:18 -07:00
Harshavardhana e18c0ab9bf update vulncheck to go1.21.9
Signed-off-by: Harshavardhana <harsha@minio.io>
2024-04-03 19:00:59 -07:00
Harshavardhana d7daae4762
update to latest deps (#19399)
Signed-off-by: Harshavardhana <harsha@minio.io>
2024-04-03 09:28:32 -07:00
Aditya Manthramurthy 7e45d84ace
ldap: improve normalization of DN values (#19358)
Instead of relying on user input values, we use the DN value returned by
the LDAP server.

This handles cases like when a mapping is set on a DN value
`uid=svc.algorithm,OU=swengg,DC=min,DC=io` with a user input value (with
unicode variation) of `uid=svc﹒algorithm,OU=swengg,DC=min,DC=io`. The
LDAP server on lookup of this DN returns the normalized value where the
unicode dot character `SMALL FULL STOP` (in the user input), gets
replaced with regular full stop.
2024-03-27 23:45:26 -07:00
Klaus Post 7ff4164d65
Fix races in IAM cache lazy loading (#19346)
Fix races in IAM cache

Fixes #19344

On the top level we only grab a read lock, but we write to the cache if we manage to fetch it.

a03dac41eb/cmd/iam-store.go (L446) is also flipped to what it should be AFAICT.

Change the internal cache structure to a concurrency safe implementation.

Bonus: Also switch grid implementation.
2024-03-26 11:12:57 -07:00
Harshavardhana 485298b680
update all dependencies (#19235) 2024-03-14 17:41:26 -07:00
dependabot[bot] 1787bcfc91
build(deps): bump github.com/lestrrat-go/jwx from 1.2.28 to 1.2.29 (#19226)
Bumps [github.com/lestrrat-go/jwx](https://github.com/lestrrat-go/jwx) from 1.2.28 to 1.2.29.
- [Release notes](https://github.com/lestrrat-go/jwx/releases)
- [Changelog](https://github.com/lestrrat-go/jwx/blob/v1.2.29/Changes)
- [Commits](https://github.com/lestrrat-go/jwx/compare/v1.2.28...v1.2.29)

---
updated-dependencies:
- dependency-name: github.com/lestrrat-go/jwx
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-08 08:42:47 -08:00
Harshavardhana 48b590e14b
fix: same server to be part of multiple pools (#19216)
our PoolNumber calculation was costly,
while we already had this information per
endpoint, we needed to deduce it appropriately.

This PR addresses this by assigning PoolNumbers
field that carries all the pool numbers that
belong to a server.

properties.PoolNumber still carries a valid value
only when len(properties.PoolNumbers) == 1, otherwise
properties.PoolNumber is set to math.MaxInt (indicating
that this value is undefined) and then one must rely
on properties.PoolNumbers for server participation
in multiple pools.

addresses the issue originating from #11327
2024-03-07 10:24:07 -08:00
Harshavardhana 5dc1ef0b87
fix: go mod update v1.33.0 https://pkg.go.dev/vuln/GO-2024-2611 (#19208) 2024-03-05 14:38:15 -08:00
Harshavardhana 6d08af61a0
for root disks add additional information in the error log (#19177) 2024-03-02 23:45:39 -08:00
Andreas Auernhammer 09626d78ff
automatically generate root credentials with KMS (#19025)
With this commit, MinIO generates root credentials automatically
and deterministically if:

 - No root credentials have been set.
 - A KMS (KES) is configured.
 - API access for the root credentials is disabled (lockdown mode).

Before, MinIO defaults to `minioadmin` for both the access and
secret keys. Now, MinIO generates unique root credentials
automatically on startup using the KMS.

Therefore, it uses the KMS HMAC function to generate pseudo-random
values. These values never change as long as the KMS key remains
the same, and the KMS key must continue to exist since all IAM data
is encrypted with it.

Backward compatibility:

This commit should not cause existing deployments to break. It only
changes the root credentials of deployments that have a KMS configured
(KES, not a static key) but have not set any admin credentials. Such
implementations should be rare or not exist at all.

Even if the worst case would be updating root credentials in mc
or other clients used to administer the cluster. Root credentials
are anyway not intended for regular S3 operations.

Signed-off-by: Andreas Auernhammer <github@aead.dev>
2024-03-01 13:09:42 -08:00
Alex ef06644799
Updated Console to v1.0.0 (#19164)
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
2024-02-29 17:54:07 -08:00
Harshavardhana d7520f0ae6
fix: make sure maintenance=true is honored properly (#19156)
fixes a regression from #18700
2024-02-29 08:37:57 -08:00
Anis Eleuch 2bdb9511bd
heal: Add skipped objects to the heal summary (#19142)
New disk healing code skips/expires objects that ILM supposed to expire.
Add more visibility to the user about this activity by calculating those
objects and print it at the end of healing activity.
2024-02-28 09:05:40 -08:00
Harshavardhana b6e98aed01
fix: found races in accessing globalLocalDrives (#19069)
make a copy before accessing globalLocalDrives

Bonus: update console v0.46.0

Signed-off-by: Harshavardhana <harsha@minio.io>
2024-02-16 17:15:57 -08:00
Harshavardhana 6fd0b434e2
upgrade all deps (#19041) 2024-02-14 09:51:34 -08:00
Anis Eleuch 4fa06aefc6
Convert service account add/update expiration to cond values (#19024)
In order to force some users allowed to create or update a service
account to provide an expiration satifying the user policy conditions.
2024-02-12 08:36:16 -08:00
Harshavardhana 7e023f2d50 remove go.mod replace tag 2024-02-06 01:56:57 -08:00
Poorna 27d02ea6f7
metrics: add replication metrics on proxied requests (#18957) 2024-02-05 22:00:45 -08:00
Anis Eleuch 6fd63e920a
log: Use error log type instead of Application/MinIO type (#18930)
* log: Use error log type instead of Application/MinIO type

Also bump github.com/shirou/gopsutil version to address cross
compilation issues.

* Apply suggestions from code review

Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>

---------

Co-authored-by: Anis Eleuch <anis@min.io>
Co-authored-by: Harshavardhana <harsha@minio.io>
Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>
2024-02-01 16:13:57 -08:00
Harshavardhana 88837fb753
add new update v2 that updates per node, allows idempotent behavior (#18859)
add new update v2 that updates per node, allows idempotent behavior

new API ensures that

- binary is correct and can be downloaded checksummed verified
- committed to actual path
- restart returns back the relevant waiting drives
2024-01-26 08:40:13 -08:00
Harshavardhana e11d851aee
add new drive I/O waiting/tokens metric (#18836)
Bonus: add virtual memory used as well part of the system resource metrics.
2024-01-19 14:51:36 -08:00
Harshavardhana 099e88516d
update all dependencies and console v0.45.0 (#18773) 2024-01-10 23:46:16 -08:00
Daniel Valdivia 5fc7da345d
Upgrade Console to v0.44.0 (#18717)
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
2023-12-27 11:19:13 -08:00
Harshavardhana fd2c38fbef
upgrade Go mod dependencies (#18710) 2023-12-25 01:02:59 -08:00
Anis Eleuch 22f8e39b58
tier: Allow edit of the new Azure and AWS auth params (#18690)
Allow editing for the service principal credentials from Azure
and the web identity token for AWS;

Also, more validation of input parameters.
2023-12-21 16:58:10 -08:00
Cesar N 7a311a3b66
Update Console version to v0.43.1 (#18683) 2023-12-19 17:00:02 -08:00
Harshavardhana f4389fb322
update dependencies for https://pkg.go.dev/vuln/GO-2023-2402 (#18676) 2023-12-18 18:35:20 -08:00
Harshavardhana bdf4e386cf
update go dependencies (#18662) 2023-12-15 21:35:07 -08:00
Krishnan Parthasarathi 162eced7d2
Fix incorrect metric desc for bucketRequestsDuration (#18657) 2023-12-14 19:02:11 -08:00
Anis Eleuch 8771617199
tier: Add support of AWS S3 tiering with web identity token file (#18648) 2023-12-14 14:01:49 -08:00
Harshavardhana 8fa2898ff1
update console UI to v0.43.0 release (#18653) 2023-12-14 10:50:54 -08:00
Anis Eleuch 4a21dce2b5
tier: Add support of SP credentials with Azure (#18630)
Co-authored-by: Anis Elleuch <anis@min.io>
2023-12-11 21:51:53 -08:00
Anis Eleuch 2e23e61a45
Add support of conf file to pass arguments and options (#18592) 2023-12-07 01:33:56 -08:00
Alex cfed671ea3
Update Console version to v0.42.1 (#18606)
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
2023-12-06 20:16:00 -08:00
Shireesh Anjal 7350a29fec
Capture percentage of cpu load and memory used (#18596)
By default the cpu load is the cumulative of all cores. Capture the
percentage load (load * 100 / cpu-count)

Also capture the percentage memory used (used * 100 / total)
2023-12-06 13:19:59 -08:00
Harshavardhana 73dde66dbe
stick to go1.19 go.mod (#18600) 2023-12-06 01:09:22 -08:00
Klaus Post 94fbcd8ebe
Add TLS cert checksum (#18557)
It allows validation of whether all certs match across clusters.
2023-11-30 12:13:50 -08:00
Harshavardhana e9babf3dac
(chore): update all our deps (#18525) 2023-11-28 14:44:44 -08:00