Changelog
- [Changed] Allow users select files and directories and keep selection whenever they navigate to multiple directories or use filter (#3527)
- [Fixed] Fixed adding metadata for S3 entries ([#3367]https://github.com/quiltdata/quilt/pull/3367)
- [Fixed] Fix editing nested files in packages, fix editing files added from the different location to package (#3117)
- [Changed] Rework package indexing: now package indexes have documents only for current versions of package pointer objects, documents for 'latest' pointers have
package_hash
,package_stats
,comment
,metadata
fields properly populated (#2897) - [Changed] Fixed config and docs mistyping:
ui.athena.defaultWorkflow
should beui.athena.defaultWorkgroup
(#3067) - [Changed] Use dedicated columns field instead of first row, fix duplicated first row in table results (#3101)
- [Changed] Allow pushing empty packages, suggest creating a stub
README.md
file when trying to push an empty package (#3114) - [Changed] Initialize Catalog configuration synchronously from
QUILT_CATALOG_CONFIG
global var (#3166)
- [Added] Declared compatibility with
jsonschema==4.*
. - [Added]
--host
and--port
parameters forquilt3 catalog
. - [Added]
--no-browser
parameter forquilt3 catalog
. - [Changed]
quilt3 catalog
now requiresquilt3
to be installed withcatalog
extra dependency (pip install 'quilt3[catalog]'
). - [Changed] Dependencies on
flask
anddnspython
are dropped. - [Removed] Deprecated passing subpackage as part of package name for
Package.install()
, usepath
parameter instead. - [Removed] Deprecated calling of
Package.resolve_hash()
without specifyingname
parameter. - [Removed] Deprecated
PackageEntry.physical_keys
, usePackageEntry.physical_key
instead.
- [Removed] Deprecated passing subpackage as part of package name for
quilt3 install
, use--path
parameter instead.
- [Changed]
pkgpush
lambda now uses dedicated lambda for hashing files to push larger packages faster from catalog.
- [Added] Size of each manifest record is now limited by 1 MB. This constraint is added to ensure that S3 select, Athena and downstream services work correctly. This limit can be overridden with
QUILT_MANIFEST_MAX_RECORD_SIZE
environment variable. (#2114)
- [Changed] Scan more bytes (first 128 KiB) when trying to detect if a JSON file is a Vega visualization (#2229)
- [Changed] Enhance
quilt_summarize.json
format, support title, description and multi-column layout (#2245) - [Changed] Remove custom button for adding Readme, and re-use dialog for creating files. Fix creating README in package (#3173)
- [Fixed] Catch and display package-related errors properly ("no such package" and "bad revision") (#2160)
- [Fixed] lambda previews for time series
AICSImage
data (potentialIndexError
if odd number of time points) (#1945) - [Fixed] Use EncodingType=url for S3 list requests to handle special chars in keys / prefixes (#2026)
- [Fixed] Empty response from
pkgselect
folder view for packages with all non-string logical_keys or physical_keys (#1947) - [Fixed] "Download Directory" bug that caused 502 or failed downloads when any files contained the "+" character (#2067)
- [Added] PDF and other file formats, including .gz ones, to catalog Overview tab
- Refactors local and s3 storage-layer code around a new PackageRegistry base class (to support improved file layouts in future releases)
- Multi-threaded download for large files, 2X to 5X performance gains when installing packages with large files, especially on larger EC2 instances
- Package name added to Package.resolve_hash
- Bugfix: remove package revision by shorthash
- Performance improvements for build and push
- PDF previews
- Browse full package contents (no longer limited to 1000 files)
- Indexing and search package-level metadata
- Fixed issue with download button for certain text files
- FCS files: content indexing and preview
- Catalog sign-in with email (or username)
- Catalog support for sign-in with Okta
- .cef preview
- allow hiding download button
- only show stats for 2-level extensions for .gz files
quilt3.logged_in()
- fix retries during hashing
- improve progress bars
- fix
quilt3 catalog
- expanded documentation
- reduce
pyyaml
requirements to prevent version conflicts
- improve unit test coverage for indexing lambdas
- fix real-time delete handling (incl. for unversioned objects)
- handle all s3:ObjectCreated: and ObjectRemoved: events (fixes ES search state and bucket Overview)
- Official support for Windows
- Add support for Python 3.7, 3.8
- Fix Package import in Python
- Updated libraries for stability and security
- Quiet TQDM for log files ($ export QUILT_MINIMIZE_STDOUT=true )
- CLI setting of config parameters
- new feature to filter large S3 directories with regex
- more reliable bucket region inference
- Support preview of larger Jupyter notebooks in S3 (via transparent GZIP)
- JS (catalog) dependencies for stability and security
- extended Parquet file support (for files without a .parquet extension)
- Improvements to catalog signing logic for external and in-stack buckets
Special thanks to @NathanDeMaria (CLI and Windows support) and @JacksonMaxfield for contributing code to this release.
Python
- Add
push
to CLI
Catalog
- Updated JS dependencies
- Display package truncation warning in Packages
Python
quilt3 install foo/bar/subdirectory
- Bug fixes for CopyObject and other exceptions
- Fix bug introduced in 3.1.9 where uploads fail due to incorrect error checking after a HEAD request to see if an object already exists (#1512)
quilt3 install
now displays the tophash of the installed package (#1461)- Added
quilt3 --version
(#1495) - Added
quilt3 disable-telemetry
CLI command (#1496) - CLI command to launch catalog directly to file viewer -
quilt3 catalog $S3_URL
(#1470, #1487) - No longer run local container for
quilt3 catalog
(#1504). See (#1468, #1483, #1482) for various bugs leading to this decision. - Add PhysicalKey class to abstract away local files vs unversioned s3 object vs versioned s3 object (#1456, #1473, #1478)
- Changed cache directory location (#1466)
- More informative progress bars (#1506)
- Improve support for downloading from public buckets (#1503)
- Always disable telemetry during tests (#1494)
- Bug fix: prevent misleading CLI argument abbreviations (#1481) such as
--to
referring to--tophash
- Bug fix: background upload/download threads are now killed if the main thread is interrupted (#1486)
- Performance improvements: load JSONL manifest faster (#1480)
- Performance improvement: If there is an error when copying files, fail quickly (#1488)
- Better package listing UX (#1462)
- Improve bucket stats visualization when there are many categories (#1469)
- Bug-fixes for
quilt3.config
andquilt3.catalog
- Performance improvements for Packages
- Updated landing page
- New
LOCAL
mode for running the catalog on localhost
quilt3 catalog
command to run the Quilt catalog on your local machinequilt3 verify
compares the state of a directory to the contents of a package version- Added a local file cache for installed packages
- Performance improvements for upload and download
- Support for short hashes to identify package versions
- Adding telemetry for API calls
- Implement Package.rollback
- Drop support for object metadata (outside of packages)
- Change the number of threads used when installing and pushing from 4 to 10 (S3 default)
- Misc bug fixes
- Fix package listing for packages with more 100 revisions
- Add stacked area charts for downloads