Changelog
- [Fixed] Fix editing nested files in packages, fix editing files added from the different location to package (#3117)
- [Changed] Rework package indexing: now package indexes have documents only for current versions of package pointer objects, documents for 'latest' pointers have
package_hash,package_stats,comment,metadatafields properly populated (#2897) - [Changed] Fixed config and docs mistyping:
ui.athena.defaultWorkflowshould beui.athena.defaultWorkgroup(#3067) - [Changed] Use dedicated columns field instead of first row, fix duplicated first row in table results (#3101)
- [Changed] Allow pushing empty packages, suggest creating a stub
README.mdfile when trying to push an empty package (#3114) - [Changed] Initialize Catalog configuration synchronously from
QUILT_CATALOG_CONFIGglobal var (#3166)
- [Added] Declared compatibility with
jsonschema==4.*. - [Added]
--hostand--portparameters forquilt3 catalog. - [Added]
--no-browserparameter forquilt3 catalog. - [Changed]
quilt3 catalognow requiresquilt3to be installed withcatalogextra dependency (pip install quilt3[catalog]). - [Changed] Dependencies on
flaskanddnspythonare dropped. - [Removed] Deprecated passing subpackage as part of package name for
Package.install(), usepathparameter instead. - [Removed] Deprecated calling of
Package.resolve_hash()without specifyingnameparameter. - [Removed] Deprecated
PackageEntry.physical_keys, usePackageEntry.physical_keyinstead.
- [Removed] Deprecated passing subpackage as part of package name for
quilt3 install, use--pathparameter instead.
- [Changed]
pkgpushlambda now uses dedicated lambda for hashing files to push larger packages faster from catalog.
- [Added] Size of each manifest record is now limited by 1 MB. This constraint is added to ensure that S3 select, Athena and downstream services work correctly. This limit can be overridden with
QUILT_MANIFEST_MAX_RECORD_SIZEenvironment variable. (#2114)
- [Changed] Scan more bytes (first 128 KiB) when trying to detect if a JSON file is a Vega visualization (#2229)
- [Changed] Enhance
quilt_summarize.jsonformat, support title, description and multi-column layout (#2245) - [Changed] Remove custom button for adding Readme, and re-use dialog for creating files. Fix creating README in package (#3173)
- [Fixed] Catch and display package-related errors properly ("no such package" and "bad revision") (#2160)
- [Fixed] lambda previews for time series
AICSImagedata (potentialIndexErrorif odd number of time points) (#1945) - [Fixed] Use EncodingType=url for S3 list requests to handle special chars in keys / prefixes (#2026)
- [Fixed] Empty response from
pkgselectfolder view for packages with all non-string logical_keys or physical_keys (#1947) - [Fixed] "Download Directory" bug that caused 502 or failed downloads when any files contained the "+" character (#2067)
- [Fixed] Crash during load of package manifest with unicode symbols with non-unicode locale set (#1931)
- [Fixed] Bump minimum required version of tqdm. Fixes a crash (
UnseekableStreamError) during upload retry. (#1853)
- [Added] PDF and other file formats, including .gz ones, to catalog Overview tab
- Refactors local and s3 storage-layer code around a new PackageRegistry base class (to support improved file layouts in future releases)
- Multi-threaded download for large files, 2X to 5X performance gains when installing packages with large files, especially on larger EC2 instances
- Package name added to Package.resolve_hash
- Bugfix: remove package revision by shorthash
- Performance improvements for build and push
- PDF previews
- Browse full package contents (no longer limited to 1000 files)
- Indexing and search package-level metadata
- Fixed issue with download button for certain text files
- FCS files: content indexing and preview
- Catalog sign-in with email (or username)
- Catalog support for sign-in with Okta
- .cef preview
- allow hiding download button
- only show stats for 2-level extensions for .gz files
quilt3.logged_in()- fix retries during hashing
- improve progress bars
- fix
quilt3 catalog - expanded documentation
- reduce
pyyamlrequirements to prevent version conflicts
- improve unit test coverage for indexing lambdas
- fix real-time delete handling (incl. for unversioned objects)
- handle all s3:ObjectCreated: and ObjectRemoved: events (fixes ES search state and bucket Overview)
- Official support for Windows
- Add support for Python 3.7, 3.8
- Fix Package import in Python
- Updated libraries for stability and security
- Quiet TQDM for log files ($ export QUILT_MINIMIZE_STDOUT=true )
- CLI setting of config parameters
- new feature to filter large S3 directories with regex
- more reliable bucket region inference
- Support preview of larger Jupyter notebooks in S3 (via transparent GZIP)
- JS (catalog) dependencies for stability and security
- extended Parquet file support (for files without a .parquet extension)
- Improvements to catalog signing logic for external and in-stack buckets
Special thanks to @NathanDeMaria (CLI and Windows support) and @JacksonMaxfield for contributing code to this release.
Python
- Add
pushto CLI
Catalog
- Updated JS dependencies
- Display package truncation warning in Packages
Python
quilt3 install foo/bar/subdirectory- Bug fixes for CopyObject and other exceptions
- Fix bug introduced in 3.1.9 where uploads fail due to incorrect error checking after a HEAD request to see if an object already exists (#1512)
quilt3 installnow displays the tophash of the installed package (#1461)- Added
quilt3 --version(#1495) - Added
quilt3 disable-telemetryCLI command (#1496) - CLI command to launch catalog directly to file viewer -
quilt3 catalog $S3_URL(#1470, #1487) - No longer run local container for
quilt3 catalog(#1504). See (#1468, #1483, #1482) for various bugs leading to this decision. - Add PhysicalKey class to abstract away local files vs unversioned s3 object vs versioned s3 object (#1456, #1473, #1478)
- Changed cache directory location (#1466)
- More informative progress bars (#1506)
- Improve support for downloading from public buckets (#1503)
- Always disable telemetry during tests (#1494)
- Bug fix: prevent misleading CLI argument abbreviations (#1481) such as
--toreferring to--tophash - Bug fix: background upload/download threads are now killed if the main thread is interrupted (#1486)
- Performance improvements: load JSONL manifest faster (#1480)
- Performance improvement: If there is an error when copying files, fail quickly (#1488)
- Better package listing UX (#1462)
- Improve bucket stats visualization when there are many categories (#1469)
- Bug-fixes for
quilt3.configandquilt3.catalog - Performance improvements for Packages
- Updated landing page
- New
LOCALmode for running the catalog on localhost
quilt3 catalogcommand to run the Quilt catalog on your local machinequilt3 verifycompares the state of a directory to the contents of a package version- Added a local file cache for installed packages
- Performance improvements for upload and download
- Support for short hashes to identify package versions
- Adding telemetry for API calls
- Implement Package.rollback
- Drop support for object metadata (outside of packages)
- Change the number of threads used when installing and pushing from 4 to 10 (S3 default)
- Misc bug fixes
- Fix package listing for packages with more 100 revisions
- Add stacked area charts for downloads
- 2-level file-extensions for bucket summary
- Fix uploads of very large files
- Remove unnecessary copying during push
- Bug fix: when adding python objects to a package a temporary file would be created and then deleted when the object was pushed, leading to a crash if you tried to push that package again (PR #1264)
- Added support for adding an in-memory object (such as a
pandas.DataFrame) to a package viapackage.set() - Fix to work with pyarrow 0.15.0
- Performance improvements for list_packages and delete_package
- Added
list_package_versionsfunction
This is the initial release of the new and improved Quilt 3 (formerly Quilt T4). For more information refer to the documentation.
Adds a feature to allow
quilt config to set a registry URL for a private Teams registry.- Adding a hash argument to
quilt.pushto allow pushing any package version to a registry.
- Make object sizes required.
- Update urllib3 version for security patch
- Improved instructions for running registries.
- Fix an ascii decoding issue related to ellipses …
- Update Parquet reading code to match the API change in pyarrow 0.11.
- Fix downloading of zero-byte files
- New helper function
quilt.saveadds an object (e.g., a Pandas DataFrame) to an existing package by performing a sub-package build and push in a single step - BugFix:
quilt.loadnow correctly returns sub-packages (fixes issue #741)
- Send a welcome email to new users after activation
- fixes an issue with packages created on older versions of pyarrow
- improves readability for
quilt inspect - allow adding a node with metadata using sub-package build/push
- adds documentation for running a private registry in AWS
- Suppress numpy warnings under Python 2.7
- Fix subpackage build and push
- Added support for sub-package build and push to allow updates to allow adding nodes to large packages without materializing the whole package
- First-class support for
ndarray
- Replaced dependence on external OAuth2 provider with a built-in authentication and session management
- Registry support for sub-package push
- Updated to support new registry authentication
- added Bracket accessor for GroupNodes
- asa.plot to show images in packages
- asa.torch to convert packages to PyTorch Datasets
- Enforce fragment store as read-only
- Added source maps and CI for catalog testing
Expands and improves documentation for working with Quilt packages.
- Load packages by hash
- Choose a custom loader for DataNodes with asa=
- Specify Ubuntu version in Dockerfiles
- display package traffic stats in catalog
- filter packages based on per-node metadata
- get/set metadata for package nodes
- support custom loaders in the _data method
- package commenting
- Metadata-only package install
- Build DataFrames from existing Parquet files
- Remove HDF5 dependencies
- Code cleanup and refactoring